2024-04-24T16:10:29.291 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T16:10:29.291 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778 --verbose --owner scheduled_adking@teuthology --name adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} -- /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778/orig.config.yaml 2024-04-24T16:10:29.339 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T16:10:29.430 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778 branch: wip-adk-testing-2024-04-24-0937 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: adking@redhat.com first_in_suite: false job_id: '7671778' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-04-24-0937 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 5904d29475f5be602879d9fb26280e89b808d5cc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5904d29475f5be602879d9fb26280e89b808d5cc workunit: branch: wip-adk-testing-2024-04-24-0937 sha1: 5904d29475f5be602879d9fb26280e89b808d5cc owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 2532 sha1: 5904d29475f5be602879d9fb26280e89b808d5cc sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-04-24-0937 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5904d29475f5be602879d9fb26280e89b808d5cc targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDmJfBoZw+B1CZPo3ctZgRjX+PElSCyGtxZy011gg5DYYmbCARINLPaU2BHCI5ySvixRHiNMxl+eLMyyXOWKeDM= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPn+3/er7ip06BTa3GEvE4tISGx6yi1N/Lc7LwpQJH3ELMqK+xGcZUt7/Yriq2YQNSLq6oIW1WyotbW62qtSStU= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_15:49:18 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T16:10:29.430 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa; will attempt to use it 2024-04-24T16:10:29.431 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks 2024-04-24T16:10:29.431 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T16:10:29.433 INFO:teuthology.task.internal:Checking packages... 2024-04-24T16:10:29.457 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5904d29475f5be602879d9fb26280e89b808d5cc' 2024-04-24T16:10:29.457 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T16:10:29.457 INFO:teuthology.packaging:ref: None 2024-04-24T16:10:29.457 INFO:teuthology.packaging:tag: None 2024-04-24T16:10:29.457 INFO:teuthology.packaging:branch: wip-adk-testing-2024-04-24-0937 2024-04-24T16:10:29.457 INFO:teuthology.packaging:sha1: 5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:10:29.458 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-04-24-0937 2024-04-24T16:10:29.701 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3285.g5904d294 2024-04-24T16:10:29.702 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T16:10:29.709 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T16:10:29.710 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T16:10:29.717 INFO:teuthology.task.internal:Saving configuration 2024-04-24T16:10:29.735 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T16:10:29.741 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T16:10:29.762 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 16:05:24.305906', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDmJfBoZw+B1CZPo3ctZgRjX+PElSCyGtxZy011gg5DYYmbCARINLPaU2BHCI5ySvixRHiNMxl+eLMyyXOWKeDM='} 2024-04-24T16:10:29.784 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 16:05:24.304550', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPn+3/er7ip06BTa3GEvE4tISGx6yi1N/Lc7LwpQJH3ELMqK+xGcZUt7/Yriq2YQNSLq6oIW1WyotbW62qtSStU='} 2024-04-24T16:10:29.785 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T16:10:29.791 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-24T16:10:29.791 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-24T16:10:29.791 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T16:10:29.863 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe26a0fc160>, signals=[15]) 2024-04-24T16:10:29.864 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T16:10:29.870 INFO:teuthology.task.internal:Opening connections... 2024-04-24T16:10:29.870 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-04-24T16:10:29.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T16:10:29.947 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T16:10:29.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T16:10:30.022 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T16:10:30.028 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-04-24T16:10:30.046 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-04-24T16:10:30.046 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-04-24T16:10:30.103 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-04-24T16:10:30.103 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-04-24T16:10:30.103 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-04-24T16:10:30.103 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T16:10:30.104 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T16:10:30.105 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-04-24T16:10:30.127 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-04-24T16:10:30.145 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-04-24T16:10:30.145 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-04-24T16:10:30.202 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-04-24T16:10:30.202 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-04-24T16:10:30.202 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-04-24T16:10:30.202 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T16:10:30.203 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T16:10:30.203 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-04-24T16:10:30.227 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T16:10:30.236 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T16:10:30.241 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T16:10:30.241 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-04-24T16:10:30.244 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-04-24T16:10:30.259 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T16:10:30.265 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T16:10:30.265 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-04-24T16:10:30.300 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-04-24T16:10:30.326 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T16:10:30.379 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T16:10:30.394 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T16:10:30.394 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T16:10:30.395 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T16:10:30.395 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T16:10:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:10:30.411 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-04-24T16:10:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:10:30.412 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-24T16:10:30.467 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-437.el9.x86_64 2024-04-24T16:10:30.468 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.14.0-437.el9.x86_64 2024-04-24T16:10:30.468 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-04-24T16:10:30.469 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-437.el9.x86_64 2024-04-24T16:10:30.469 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-437.el9.x86_64 2024-04-24T16:10:30.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-04-24T16:10:33.142 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-04-24T16:10:34.496 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-04-24T16:10:36.117 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 19 MB 00:01 2024-04-24T16:10:40.351 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-04-24T16:10:42.118 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-24T16:10:43.311 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-24T16:10:48.187 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-04-24T16:10:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-04-24T16:10:50.258 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T16:10:50.259 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T16:10:50.355 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-24T16:10:50.356 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-04-24T16:10:50.356 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-24T16:10:50.473 DEBUG:teuthology.orchestra.run.smithi070:> echo no | sudo yum reinstall kernel || true 2024-04-24T16:10:50.967 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Apr 2024 04:10:48 PM UTC. 2024-04-24T16:10:51.086 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-24T16:10:51.094 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout:Reinstalling: 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 255 k 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 0 2024-04-24T16:10:51.095 INFO:teuthology.orchestra.run.smithi070.stderr:Operation aborted. 2024-04-24T16:10:51.156 INFO:teuthology.orchestra.run.smithi070.stdout:Is this ok [y/N]: 2024-04-24T16:10:51.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum reinstall -y kernel || true 2024-04-24T16:10:51.651 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 04:10:48 PM UTC. 2024-04-24T16:10:51.767 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:Reinstalling: 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 255 k 2024-04-24T16:10:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 0 2024-04-24T16:10:51.769 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-24T16:10:52.579 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.2 MB 00:00 2024-04-24T16:10:52.604 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-437.el9.x86_64.rpm 536 kB/s | 255 kB 00:00 2024-04-24T16:10:52.604 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-24T16:10:52.604 INFO:teuthology.orchestra.run.smithi070.stdout:Total 305 kB/s | 255 kB 00:00 2024-04-24T16:10:52.620 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-24T16:10:52.796 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-24T16:10:52.796 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-24T16:10:52.801 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-24T16:10:52.801 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-24T16:10:52.934 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-24T16:10:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T16:10:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T16:10:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T16:10:53.631 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T16:10:54.155 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-24T16:10:54.334 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T16:10:54.334 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:10:54.334 INFO:teuthology.orchestra.run.smithi070.stdout:Reinstalled: 2024-04-24T16:10:54.334 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T16:10:54.334 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:10:54.334 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-24T16:10:54.525 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T16:10:54.572 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T16:10:54.572 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T16:10:54.572 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T16:10:54.572 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T16:10:54.572 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T16:10:54.605 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-04-24T16:10:54.625 DEBUG:teuthology.parallel:result is None 2024-04-24T16:10:55.223 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-24T16:11:00.231 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-04-24T16:11:02.087 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T16:11:02.094 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T16:11:02.170 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-24T16:11:02.171 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-04-24T16:11:02.171 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-24T16:11:02.278 DEBUG:teuthology.orchestra.run.smithi156:> echo no | sudo yum reinstall kernel || true 2024-04-24T16:11:02.820 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Apr 2024 04:11:00 PM UTC. 2024-04-24T16:11:02.940 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-24T16:11:02.948 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T16:11:02.948 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-04-24T16:11:02.948 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T16:11:02.948 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalling: 2024-04-24T16:11:02.948 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T16:11:02.948 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:11:02.948 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-24T16:11:02.949 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T16:11:02.949 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:11:02.949 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 255 k 2024-04-24T16:11:02.949 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 0 2024-04-24T16:11:02.949 INFO:teuthology.orchestra.run.smithi156.stderr:Operation aborted. 2024-04-24T16:11:03.016 INFO:teuthology.orchestra.run.smithi156.stdout:Is this ok [y/N]: 2024-04-24T16:11:03.016 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum reinstall -y kernel || true 2024-04-24T16:11:03.514 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 04:11:00 PM UTC. 2024-04-24T16:11:03.631 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-24T16:11:03.631 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalling: 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 255 k 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 0 2024-04-24T16:11:03.632 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-04-24T16:11:04.683 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-437.el9.x86_64.rpm 258 kB/s | 255 kB 00:00 2024-04-24T16:11:04.684 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-04-24T16:11:04.684 INFO:teuthology.orchestra.run.smithi156.stdout:Total 242 kB/s | 255 kB 00:01 2024-04-24T16:11:04.705 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-04-24T16:11:04.910 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-04-24T16:11:04.910 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-04-24T16:11:04.915 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-04-24T16:11:04.915 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-04-24T16:11:05.070 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-04-24T16:11:05.159 INFO:teuthology.orchestra.run.smithi156.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T16:11:05.159 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T16:11:05.782 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T16:11:05.782 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T16:11:06.497 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T16:11:06.498 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:11:06.498 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalled: 2024-04-24T16:11:06.498 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T16:11:06.498 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:11:06.498 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-24T16:11:06.688 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T16:11:06.734 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T16:11:06.735 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T16:11:06.736 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T16:11:06.736 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T16:11:06.736 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T16:11:06.769 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2024-04-24T16:11:06.794 DEBUG:teuthology.parallel:result is None 2024-04-24T16:11:06.794 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T16:11:06.802 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T16:11:06.802 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T16:11:06.805 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T16:11:06.849 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T16:11:06.856 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T16:11:06.866 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T16:11:06.866 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T16:11:06.869 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T16:11:06.917 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T16:11:06.924 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T16:11:06.924 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T16:11:06.936 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T16:11:06.989 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T16:11:07.000 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T16:11:07.014 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T16:11:07.023 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T16:11:07.024 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T16:11:07.031 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T16:11:07.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T16:11:07.034 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T16:11:07.058 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T16:11:07.067 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T16:11:07.067 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T16:11:07.101 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T16:11:07.116 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T16:11:07.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T16:11:07.252 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T16:11:07.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T16:11:07.377 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:11:07.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T16:11:07.443 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T16:11:07.482 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T16:11:07.527 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T16:11:07.612 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T16:11:07.649 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T16:11:07.649 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T16:11:07.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-04-24T16:11:07.715 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-04-24T16:11:07.756 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T16:11:07.799 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T16:11:08.193 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T16:11:08.202 INFO:teuthology.task.internal:Starting timer... 2024-04-24T16:11:08.202 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T16:11:08.212 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T16:11:08.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-04-24T16:11:08.296 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-04-24T16:11:08.299 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-04-24T16:11:08.344 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-04-24T16:11:08.346 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T16:11:08.346 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-04-24T16:11:08.370 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-04-24T16:11:08.370 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-04-24T16:11:08.409 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-04-24T16:11:08.409 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2024-04-24T16:11:08.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T16:11:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:11:08.436 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T16:11:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:11:08.473 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T16:11:08.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-04-24T16:11:08.500 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-04-24T16:11:08.546 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T16:11:08.560 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T16:11:08.565 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T16:11:08.701 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T16:11:08.718 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T16:11:08.719 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T16:14:35.226 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2024-04-24T16:14:35.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-24T16:14:35.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T16:14:35.301 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-24T16:14:35.381 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-24T16:14:35.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-24T16:14:35.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T16:14:35.454 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-04-24T16:14:35.532 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-24T16:14:35.533 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T16:14:35.543 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T16:14:35.544 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T16:14:35.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T16:14:35.546 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T16:14:35.546 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T16:14:35.571 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T16:14:35.584 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T16:14:35.600 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T16:14:35.606 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-04-24T16:14:35.613 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T16:14:35.616 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-04-24T16:14:35.629 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T16:14:35.636 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-04-24T16:14:35.641 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T16:14:35.646 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-04-24T16:14:35.659 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T16:14:35.672 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T16:14:35.702 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-24T16:14:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T16:14:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-24T16:14:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.706 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.727 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-24T16:14:35.730 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T16:14:35.730 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-24T16:14:35.730 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.730 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.731 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.731 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:14:35.731 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-24T16:14:35.817 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '5904d29475f5be602879d9fb26280e89b808d5cc'} 2024-04-24T16:14:35.817 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-24T16:14:35.818 INFO:tasks.cephadm:Cluster fsid is be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:14:35.818 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-24T16:14:35.818 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.70', 'mon.c': '[v2:172.21.15.70:3301,v1:172.21.15.70:6790]', 'mon.b': '172.21.15.156'} 2024-04-24T16:14:35.818 INFO:tasks.cephadm:First mon is mon.a on smithi070 2024-04-24T16:14:35.818 INFO:tasks.cephadm:First mgr is y 2024-04-24T16:14:35.818 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-24T16:14:35.818 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2024-04-24T16:14:35.843 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2024-04-24T16:14:35.867 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-24T16:14:35.867 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T16:14:36.145 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 24 16:14 /home/ubuntu/cephtest/cephadm 2024-04-24T16:14:36.145 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T16:14:36.434 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 24 16:14 /home/ubuntu/cephtest/cephadm 2024-04-24T16:14:36.434 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T16:14:36.450 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T16:14:36.474 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-24T16:14:36.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-24T16:14:36.492 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-24T16:14:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-24T16:14:38.086 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-24T16:14:58.212 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-24T16:14:58.213 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-24T16:14:58.213 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-24T16:14:58.213 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2024-04-24T16:14:58.213 INFO:teuthology.orchestra.run.smithi156.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-24T16:14:58.213 INFO:teuthology.orchestra.run.smithi156.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-24T16:14:58.213 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-04-24T16:14:58.213 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-24T16:14:58.703 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:14:58.703 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-24T16:14:58.703 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-24T16:14:58.704 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2024-04-24T16:14:58.704 INFO:teuthology.orchestra.run.smithi070.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-24T16:14:58.704 INFO:teuthology.orchestra.run.smithi070.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-24T16:14:58.704 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-24T16:14:58.704 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:14:58.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2024-04-24T16:14:58.778 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2024-04-24T16:14:58.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2024-04-24T16:14:58.829 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2024-04-24T16:14:58.872 INFO:tasks.cephadm:Writing seed config... 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-24T16:14:58.873 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-24T16:14:58.874 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:14:58.874 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-24T16:14:58.888 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = be31be9a-0255-11ef-bc93-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-24T16:14:58.888 DEBUG:teuthology.orchestra.run.smithi070:mon.a> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service 2024-04-24T16:14:58.931 DEBUG:teuthology.orchestra.run.smithi070:mgr.y> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service 2024-04-24T16:14:58.973 INFO:tasks.cephadm:Bootstrapping... 2024-04-24T16:14:58.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid be31be9a-0255-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.70 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-24T16:14:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:-------------------------------------------------------------------------------- 2024-04-24T16:14:59.166 INFO:teuthology.orchestra.run.smithi070.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'be31be9a-0255-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.70', '--skip-admin-label'] 2024-04-24T16:14:59.189 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: 4.9.4-dev 2024-04-24T16:14:59.191 INFO:teuthology.orchestra.run.smithi070.stderr:Verifying podman|docker is present... 2024-04-24T16:14:59.212 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: 4.9.4-dev 2024-04-24T16:14:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:Verifying lvm2 is present... 2024-04-24T16:14:59.214 INFO:teuthology.orchestra.run.smithi070.stderr:Verifying time synchronization is in place... 2024-04-24T16:14:59.221 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T16:14:59.227 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: inactive 2024-04-24T16:14:59.234 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: enabled 2024-04-24T16:14:59.239 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: active 2024-04-24T16:14:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:Unit chronyd.service is enabled and running 2024-04-24T16:14:59.240 INFO:teuthology.orchestra.run.smithi070.stderr:Repeating the final host check... 2024-04-24T16:14:59.261 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: 4.9.4-dev 2024-04-24T16:14:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-24T16:14:59.263 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl is present 2024-04-24T16:14:59.264 INFO:teuthology.orchestra.run.smithi070.stderr:lvcreate is present 2024-04-24T16:14:59.269 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T16:14:59.275 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: inactive 2024-04-24T16:14:59.281 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: enabled 2024-04-24T16:14:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: active 2024-04-24T16:14:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:Unit chronyd.service is enabled and running 2024-04-24T16:14:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:Host looks OK 2024-04-24T16:14:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:Cluster fsid: be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:14:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:Acquiring lock 140035004694384 on /run/cephadm/be31be9a-0255-11ef-bc93-c7b262605968.lock 2024-04-24T16:14:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:Lock 140035004694384 acquired on /run/cephadm/be31be9a-0255-11ef-bc93-c7b262605968.lock 2024-04-24T16:14:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:Verifying IP 172.21.15.70 port 3300 ... 2024-04-24T16:14:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:Verifying IP 172.21.15.70 port 6789 ... 2024-04-24T16:14:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:Base mon IP is 172.21.15.70, final addrv is [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-04-24T16:14:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.70 metric 100 2024-04-24T16:14:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.70 metric 100 2024-04-24T16:14:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-24T16:14:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-24T16:14:59.294 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-24T16:14:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-24T16:14:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-24T16:14:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-24T16:14:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-24T16:14:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: inet6 fe80::eade:d210:b60b:aae4/64 scope link noprefixroute 2024-04-24T16:14:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-24T16:14:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:Mon IP `172.21.15.70` is in CIDR network `172.21.0.0/20` 2024-04-24T16:14:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-24T16:14:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-24T16:14:59.463 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-24T16:14:59.878 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Getting image source signatures 2024-04-24T16:14:59.878 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-24T16:14:59.878 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-24T16:14:59.878 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-24T16:14:59.878 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-24T16:14:59.878 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-24T16:14:59.878 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-24T16:15:02.149 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: Writing manifest to image destination 2024-04-24T16:15:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-24T16:15:07.444 INFO:teuthology.orchestra.run.smithi070.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-24T16:15:08.596 INFO:teuthology.orchestra.run.smithi070.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-24T16:15:08.596 INFO:teuthology.orchestra.run.smithi070.stderr:Extracting ceph user uid/gid from container image... 2024-04-24T16:15:09.029 INFO:teuthology.orchestra.run.smithi070.stderr:stat: 167 167 2024-04-24T16:15:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:Creating initial keys... 2024-04-24T16:15:09.968 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph-authtool: AQANMClm8VHGNhAABVhqOdbxt/Tkpd1imTn8kg== 2024-04-24T16:15:11.040 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph-authtool: AQAOMClmGiuxMBAAdGcSsDUSfwnKUvMvgXqmVQ== 2024-04-24T16:15:12.063 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph-authtool: AQAPMClmOELrORAArz7aNXi4FV/ddOUkFWoL6Q== 2024-04-24T16:15:12.585 INFO:teuthology.orchestra.run.smithi070.stderr:Creating initial monmap... 2024-04-24T16:15:12.977 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T16:15:12.977 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-24T16:15:12.977 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:12.977 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T16:15:14.700 INFO:teuthology.orchestra.run.smithi070.stderr:monmaptool for a [v2:172.21.15.70:3300,v1:172.21.15.70:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T16:15:14.700 INFO:teuthology.orchestra.run.smithi070.stderr:setting min_mon_release = octopus 2024-04-24T16:15:14.700 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/monmaptool: set fsid to be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:14.700 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T16:15:14.701 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:15:14.701 INFO:teuthology.orchestra.run.smithi070.stderr:Creating mon... 2024-04-24T16:15:16.298 INFO:teuthology.orchestra.run.smithi070.stderr:create mon.a on 2024-04-24T16:15:16.497 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-24T16:15:16.642 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-be31be9a-0255-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-be31be9a-0255-11ef-bc93-c7b262605968.target. 2024-04-24T16:15:16.643 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-be31be9a-0255-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-be31be9a-0255-11ef-bc93-c7b262605968.target. 2024-04-24T16:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Failed to reset failed state of unit ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-24T16:15:16.942 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Created symlink /etc/systemd/system/ceph-be31be9a-0255-11ef-bc93-c7b262605968.target.wants/ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-be31be9a-0255-11ef-bc93-c7b262605968@.service. 2024-04-24T16:15:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:17 smithi070 ceph-mon[21478]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T16:15:17.610 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:17 smithi070 bash[21459]: 0f9dd8d96184199cfa440c9a7403aea2189015f131b742b4b70e8da31de8759f 2024-04-24T16:15:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: disabled 2024-04-24T16:15:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: inactive 2024-04-24T16:15:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:firewalld.service is not enabled 2024-04-24T16:15:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:Not possible to enable service . firewalld.service is not available 2024-04-24T16:15:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for mon to start... 2024-04-24T16:15:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for mon... 2024-04-24T16:15:17.868 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:17 smithi070 systemd[1]: Started Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:15:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: cluster: 2024-04-24T16:15:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: id: be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:18.133 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: services: 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.568073s) 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: data: 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: pgs: 2024-04-24T16:15:18.134 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:18.684 INFO:teuthology.orchestra.run.smithi070.stderr:mon is available 2024-04-24T16:15:18.685 INFO:teuthology.orchestra.run.smithi070.stderr:Assimilating anything we can from ceph.conf... 2024-04-24T16:15:18.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:18 smithi070 ceph-mon[21478]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T16:15:18.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:18 smithi070 ceph-mon[21478]: monmap e1: 1 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} 2024-04-24T16:15:18.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:18 smithi070 ceph-mon[21478]: fsmap 2024-04-24T16:15:18.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:18 smithi070 ceph-mon[21478]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T16:15:18.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:18 smithi070 ceph-mon[21478]: mgrmap e1: no daemons active 2024-04-24T16:15:18.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:18 smithi070 ceph-mon[21478]: from='client.? 172.21.15.70:0/1116251320' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:15:19.188 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:19.188 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: [global] 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: fsid = be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: [mgr] 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: [osd] 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-24T16:15:19.189 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-24T16:15:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:Generating new minimal ceph.conf... 2024-04-24T16:15:19.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:19 smithi070 ceph-mon[21478]: from='client.? 172.21.15.70:0/310692212' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T16:15:19.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:19 smithi070 ceph-mon[21478]: from='client.? 172.21.15.70:0/310692212' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-24T16:15:20.945 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:20 smithi070 ceph-mon[21478]: from='client.? 172.21.15.70:0/4006395786' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:15:20.948 INFO:teuthology.orchestra.run.smithi070.stderr:Restarting the monitor... 2024-04-24T16:15:21.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:20 smithi070 systemd[1]: Stopping Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:15:21.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:21 smithi070 bash[21697]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-24T16:15:21.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:21 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a[21474]: 2024-04-24T16:15:21.350+0000 7f244c683700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:15:21.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:21 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a[21474]: 2024-04-24T16:15:21.350+0000 7f244c683700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-24T16:15:21.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:21 smithi070 podman[21710]: 2024-04-24 16:15:21.442969829 +0000 UTC m=+0.423932256 container died 0f9dd8d96184199cfa440c9a7403aea2189015f131b742b4b70e8da31de8759f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, version=8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.display-name=CentOS Stream 8, release=754, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-24T16:15:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:21 smithi070 podman[21710]: 2024-04-24 16:15:21.816467244 +0000 UTC m=+0.797429674 container cleanup 0f9dd8d96184199cfa440c9a7403aea2189015f131b742b4b70e8da31de8759f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, ceph=True, io.openshift.tags=base centos centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=) 2024-04-24T16:15:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:21 smithi070 bash[21710]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a 2024-04-24T16:15:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:21 smithi070 bash[21737]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-24T16:15:22.367 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 podman[21725]: 2024-04-24 16:15:22.067340653 +0000 UTC m=+0.625018135 container remove 0f9dd8d96184199cfa440c9a7403aea2189015f131b742b4b70e8da31de8759f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vendor=Red Hat, Inc., architecture=x86_64, io.buildah.version=1.19.8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8) 2024-04-24T16:15:22.367 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-24T16:15:22.367 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 systemd[1]: Stopped Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:15:22.367 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 systemd[1]: Starting Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:15:22.367 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 podman[21808]: 2024-04-24 16:15:22.284774461 +0000 UTC m=+0.096994594 container create d744477c6ebbebcf8d9b22a9231138e8865b3e2e902bd32dd42a577bfc068c2c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, RELEASE=HEAD, release=754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, version=8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, architecture=x86_64, name=centos-stream, vendor=Red Hat, Inc.) 2024-04-24T16:15:22.367 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 podman[21808]: 2024-04-24 16:15:22.205817548 +0000 UTC m=+0.018037635 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-24T16:15:22.562 INFO:teuthology.orchestra.run.smithi070.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-24T16:15:22.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 podman[21808]: 2024-04-24 16:15:22.49515366 +0000 UTC m=+0.307373747 container init d744477c6ebbebcf8d9b22a9231138e8865b3e2e902bd32dd42a577bfc068c2c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, vendor=Red Hat, Inc., name=centos-stream, release=754, io.openshift.tags=base centos centos-stream, distribution-scope=public, maintainer=Guillaume Abrioux , version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T16:15:22.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 podman[21808]: 2024-04-24 16:15:22.497969686 +0000 UTC m=+0.310189767 container start d744477c6ebbebcf8d9b22a9231138e8865b3e2e902bd32dd42a577bfc068c2c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, name=centos-stream, version=8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, ceph=True, com.redhat.component=centos-stream-container, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-24T16:15:22.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T16:15:22.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-24T16:15:22.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: pidfile_write: ignore empty --pid-file 2024-04-24T16:15:22.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: load: jerasure load: lrc 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: RocksDB version: 6.15.5 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Compile date Apr 18 2022 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: DB SUMMARY 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: DB Session ID: W9BU06HRK2JBZFILDE7V 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: CURRENT file: CURRENT 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82627 ; 2024-04-24T16:15:22.688 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.error_if_exists: 0 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.create_if_missing: 0 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.paranoid_checks: 1 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.env: 0x56495fcc6860 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.fs: Posix File System 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.info_log: 0x56496151fdc0 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.statistics: (nil) 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.use_fsync: 0 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_log_file_size: 0 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T16:15:22.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.allow_fallocate: 1 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.use_direct_reads: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.db_log_dir: 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T16:15:22.690 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T16:15:22.691 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.write_buffer_manager: 0x5649616102a0 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T16:15:22.692 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.unordered_write: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.row_cache: None 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.wal_filter: None 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.preserve_deletes: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.two_write_queues: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.atomic_flush: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T16:15:22.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.log_readahead_size: 0 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_background_jobs: 2 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_background_compactions: -1 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_subcompactions: 1 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T16:15:22.694 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_open_files: -1 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_background_flushes: -1 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Compression algorithms supported: 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kZSTD supported: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kXpressCompression supported: 0 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T16:15:22.695 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kLZ4Compression supported: 1 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kBZip2Compression supported: 0 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kZlibCompression supported: 1 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: kSnappyCompression supported: 1 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.merge_operator: 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_filter: None 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T16:15:22.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5649614ebd00) 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_type: 0 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_index_type: 0 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_shortening: 1 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: hash_index_allow_collision: 1 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: checksum: 1 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: no_block_cache: 0 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache: 0x564961556170 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-04-24T16:15:22.697 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_options: 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: capacity : 536870912 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_shard_bits : 4 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: strict_capacity_limit : 0 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_compressed: (nil) 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: persistent_cache: (nil) 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size: 4096 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size_deviation: 10 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_restart_interval: 16 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_block_restart_interval: 1 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: metadata_block_size: 4096 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: partition_filters: 0 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_delta_encoding: 1 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-24T16:15:22.698 INFO:journalctl@ceph.mon.a.smithi070.stdout: whole_key_filtering: 1 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout: verify_compression: 0 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout: format_version: 4 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_index_compression: 1 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_align: 0 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression: NoCompression 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.num_levels: 7 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T16:15:22.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:15:22.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T16:15:22.701 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.arena_block_size: 4194304 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T16:15:22.702 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.table_properties_collectors: 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.inplace_update_support: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.bloom_locality: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.max_successive_merges: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T16:15:22.703 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.ttl: 2592000 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.enable_blob_files: false 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.min_blob_size: 0 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975322523235, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-24T16:15:22.704 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975322523925, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79177, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77440, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71749, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713975322, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "3c4bd05e-f157-4081-8681-49ef00aebfdb", "db_session_id": "W9BU06HRK2JBZFILDE7V"}} 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975322524258, "job": 1, "event": "recovery_finished"} 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x56496153ca80 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: DB pointer 0x56496154c000 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** DB Stats ** 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:15:22.705 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:15:22.706 INFO:journalctl@ceph.mon.a.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative compaction: 0.00 GB write, 26.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval compaction: 0.00 GB write, 26.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:15:22.707 INFO:journalctl@ceph.mon.a.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.7 0.00 0.00 1 0.000 0 0 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:15:22.708 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative compaction: 0.00 GB write, 26.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: starting mon.a rank 0 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???) e1 preinit fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???).mds e1 new map 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???).mds e1 print_map 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: e1 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-24T16:15:22.709 INFO:journalctl@ceph.mon.a.smithi070.stdout: legacy client fscid: -1 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout: No filesystems configured 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 bash[21808]: d744477c6ebbebcf8d9b22a9231138e8865b3e2e902bd32dd42a577bfc068c2c 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 systemd[1]: Started Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: monmap e1: 1 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0]} 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: fsmap 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T16:15:22.710 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:22 smithi070 ceph-mon[21827]: mgrmap e1: no daemons active 2024-04-24T16:15:23.652 INFO:teuthology.orchestra.run.smithi070.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-24T16:15:23.652 INFO:teuthology.orchestra.run.smithi070.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-24T16:15:23.652 INFO:teuthology.orchestra.run.smithi070.stderr:Creating mgr... 2024-04-24T16:15:23.653 INFO:teuthology.orchestra.run.smithi070.stderr:Verifying port 9283 ... 2024-04-24T16:15:23.823 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Failed to reset failed state of unit ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-24T16:15:23.830 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: Created symlink /etc/systemd/system/ceph-be31be9a-0255-11ef-bc93-c7b262605968.target.wants/ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-be31be9a-0255-11ef-bc93-c7b262605968@.service. 2024-04-24T16:15:23.988 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:23 smithi070 systemd[1]: Starting Ceph mgr.y for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:15:24.258 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:24 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2097728614' entity='client.admin' 2024-04-24T16:15:24.259 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 podman[22042]: 2024-04-24 16:15:24.109729096 +0000 UTC m=+0.017789663 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-24T16:15:24.259 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 podman[22042]: 2024-04-24 16:15:24.261510014 +0000 UTC m=+0.169570599 container create cbef4307b27d44b46080bf153021809553300fda247b84ff56d830405ee0bc5e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.expose-services=, ceph=True, release=754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-24T16:15:24.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 podman[22042]: 2024-04-24 16:15:24.482484509 +0000 UTC m=+0.390545052 container init cbef4307b27d44b46080bf153021809553300fda247b84ff56d830405ee0bc5e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, release=754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vcs-type=git) 2024-04-24T16:15:24.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 podman[22042]: 2024-04-24 16:15:24.485035246 +0000 UTC m=+0.393095790 container start cbef4307b27d44b46080bf153021809553300fda247b84ff56d830405ee0bc5e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, io.openshift.expose-services=, RELEASE=HEAD, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, architecture=x86_64, build-date=2022-05-03T08:36:31.336870) 2024-04-24T16:15:24.593 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: disabled 2024-04-24T16:15:24.600 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: inactive 2024-04-24T16:15:24.601 INFO:teuthology.orchestra.run.smithi070.stderr:firewalld.service is not enabled 2024-04-24T16:15:24.601 INFO:teuthology.orchestra.run.smithi070.stderr:Not possible to enable service . firewalld.service is not available 2024-04-24T16:15:24.610 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: disabled 2024-04-24T16:15:24.617 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: inactive 2024-04-24T16:15:24.618 INFO:teuthology.orchestra.run.smithi070.stderr:firewalld.service is not enabled 2024-04-24T16:15:24.618 INFO:teuthology.orchestra.run.smithi070.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-24T16:15:24.618 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for mgr to start... 2024-04-24T16:15:24.618 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for mgr... 2024-04-24T16:15:24.850 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 bash[22042]: cbef4307b27d44b46080bf153021809553300fda247b84ff56d830405ee0bc5e 2024-04-24T16:15:24.850 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 systemd[1]: Started Ceph mgr.y for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:15:24.850 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:24.633+0000 7fde39f3e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:15:24.850 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:24.772+0000 7fde39f3e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:15:25.141 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:24.873+0000 7fde39f3e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:15:25.144 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:25.144 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:15:25.144 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsid": "be31be9a-0255-11ef-bc93-c7b262605968", 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "health": { 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "checks": {}, 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mutes": [] 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum": [ 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 0 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-24T16:15:25.145 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "a" 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "monmap": { 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osdmap": { 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-24T16:15:25.146 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgmap": { 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-24T16:15:25.150 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsmap": { 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "available": false, 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modules": [ 2024-04-24T16:15:25.151 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "iostat", 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "nfs", 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "restful" 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "servicemap": { 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modified": "2024-04-24T16:15:17.566739+0000", 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-24T16:15:25.152 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:15:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:25 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2504845785' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T16:15:25.494 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:25.199+0000 7fde39f3e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:15:25.494 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:25.281+0000 7fde39f3e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:15:25.805 INFO:teuthology.orchestra.run.smithi070.stderr:mgr not available, waiting (1/15)... 2024-04-24T16:15:26.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:26.404+0000 7fde39f3e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:15:26.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:26.563+0000 7fde39f3e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:15:26.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:26.643+0000 7fde39f3e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:15:27.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:26.948+0000 7fde39f3e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:15:27.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:27.035+0000 7fde39f3e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:15:27.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:27.148+0000 7fde39f3e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:15:27.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:27.386+0000 7fde39f3e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:15:28.121 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:27.815+0000 7fde39f3e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:15:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:15:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsid": "be31be9a-0255-11ef-bc93-c7b262605968", 2024-04-24T16:15:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "health": { 2024-04-24T16:15:28.351 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-24T16:15:28.352 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "checks": {}, 2024-04-24T16:15:28.352 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mutes": [] 2024-04-24T16:15:28.352 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:28.352 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-24T16:15:28.352 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum": [ 2024-04-24T16:15:28.352 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 0 2024-04-24T16:15:28.352 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-24T16:15:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "a" 2024-04-24T16:15:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-24T16:15:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "monmap": { 2024-04-24T16:15:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:28.353 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-24T16:15:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-24T16:15:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osdmap": { 2024-04-24T16:15:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-24T16:15:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-24T16:15:28.354 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgmap": { 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-24T16:15:28.355 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-24T16:15:28.356 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-24T16:15:28.356 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-24T16:15:28.356 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-24T16:15:28.356 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-24T16:15:28.356 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-24T16:15:28.356 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:28.356 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsmap": { 2024-04-24T16:15:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-24T16:15:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-24T16:15:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-24T16:15:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "available": false, 2024-04-24T16:15:28.357 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-24T16:15:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modules": [ 2024-04-24T16:15:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "iostat", 2024-04-24T16:15:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "nfs", 2024-04-24T16:15:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "restful" 2024-04-24T16:15:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "servicemap": { 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modified": "2024-04-24T16:15:17.566739+0000", 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-24T16:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:15:28.446 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:28.353+0000 7fde39f3e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:15:28.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:28 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1727674429' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T16:15:28.741 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:28.448+0000 7fde39f3e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:15:28.944 INFO:teuthology.orchestra.run.smithi070.stderr:mgr not available, waiting (2/15)... 2024-04-24T16:15:29.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:28.743+0000 7fde39f3e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:15:29.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:28.831+0000 7fde39f3e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:15:29.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:29.230+0000 7fde39f3e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:15:29.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:29.360+0000 7fde39f3e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:15:29.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:29.450+0000 7fde39f3e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:15:30.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:29.563+0000 7fde39f3e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:15:30.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:29.648+0000 7fde39f3e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:15:30.532 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:30.181+0000 7fde39f3e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:15:30.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:30.534+0000 7fde39f3e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: Activating manager daemon y 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: mgrmap e2: y(active, starting, since 0.00284397s) 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: Manager daemon y is now available 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' 2024-04-24T16:15:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' 2024-04-24T16:15:30.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:30 smithi070 ceph-mon[21827]: from='mgr.14100 172.21.15.70:0/3494031257' entity='mgr.y' 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsid": "be31be9a-0255-11ef-bc93-c7b262605968", 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "health": { 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "checks": {}, 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mutes": [] 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum": [ 2024-04-24T16:15:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 0 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "a" 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "monmap": { 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osdmap": { 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:31.454 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgmap": { 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-24T16:15:31.455 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsmap": { 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "available": false, 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modules": [ 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "iostat", 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "nfs", 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "restful" 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:31.456 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "servicemap": { 2024-04-24T16:15:31.457 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:31.457 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modified": "2024-04-24T16:15:17.566739+0000", 2024-04-24T16:15:31.457 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:31.457 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:31.457 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-24T16:15:31.457 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:15:31.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:31 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1817793962' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T16:15:31.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:31 smithi070 ceph-mon[21827]: mgrmap e3: y(active, since 1.00591s) 2024-04-24T16:15:32.016 INFO:teuthology.orchestra.run.smithi070.stderr:mgr not available, waiting (3/15)... 2024-04-24T16:15:34.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:34 smithi070 ceph-mon[21827]: mgrmap e4: y(active, since 2s) 2024-04-24T16:15:34.705 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:34.705 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:15:34.705 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsid": "be31be9a-0255-11ef-bc93-c7b262605968", 2024-04-24T16:15:34.705 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "health": { 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "checks": {}, 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mutes": [] 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum": [ 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 0 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "a" 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "monmap": { 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-24T16:15:34.706 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osdmap": { 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgmap": { 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-24T16:15:34.707 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "fsmap": { 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "available": true, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-24T16:15:34.708 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modules": [ 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "iostat", 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "nfs", 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "restful" 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ], 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "servicemap": { 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "modified": "2024-04-24T16:15:17.566739+0000", 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "services": {} 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: }, 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-24T16:15:34.709 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:15:35.238 INFO:teuthology.orchestra.run.smithi070.stderr:mgr is available 2024-04-24T16:15:35.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:35 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/313989494' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T16:15:35.811 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:35.811 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: [global] 2024-04-24T16:15:35.811 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: fsid = be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:35.811 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-24T16:15:35.811 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: [mgr] 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: [osd] 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-24T16:15:35.812 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-24T16:15:36.336 INFO:teuthology.orchestra.run.smithi070.stderr:Enabling cephadm module... 2024-04-24T16:15:36.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:36 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/934375324' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T16:15:36.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:36 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/934375324' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-24T16:15:37.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:37 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2205497167' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-24T16:15:37.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setuser ceph since I am not root 2024-04-24T16:15:37.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setgroup ceph since I am not root 2024-04-24T16:15:37.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:37.703+0000 7f6eb5ae4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:15:38.114 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:37.830+0000 7f6eb5ae4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:15:38.114 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:37.928+0000 7f6eb5ae4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:15:38.450 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:38.242+0000 7f6eb5ae4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:15:38.450 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:38.324+0000 7f6eb5ae4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:15:38.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:38 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2205497167' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-24T16:15:38.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:38 smithi070 ceph-mon[21827]: mgrmap e5: y(active, since 7s) 2024-04-24T16:15:38.815 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:15:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-24T16:15:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "available": true, 2024-04-24T16:15:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-24T16:15:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-24T16:15:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:15:39.373 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for the mgr to restart... 2024-04-24T16:15:39.373 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for mgr epoch 5... 2024-04-24T16:15:39.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:39 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3663377182' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T16:15:39.633 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:39.466+0000 7f6eb5ae4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:15:39.633 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:39.634+0000 7f6eb5ae4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:15:40.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:39.722+0000 7f6eb5ae4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:15:40.482 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:40 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:40.049+0000 7f6eb5ae4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:15:40.482 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:40 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:40.143+0000 7f6eb5ae4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:15:40.482 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:40 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:40.239+0000 7f6eb5ae4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:15:40.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:40 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:40.484+0000 7f6eb5ae4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:15:41.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:40 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:40.927+0000 7f6eb5ae4000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:15:41.778 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:41 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:41.402+0000 7f6eb5ae4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:15:41.778 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:41 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:41.492+0000 7f6eb5ae4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:15:41.778 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:41 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:41.778+0000 7f6eb5ae4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:15:42.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:41 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:41.866+0000 7f6eb5ae4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:15:42.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:42 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:42.261+0000 7f6eb5ae4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:15:42.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:42 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:42.390+0000 7f6eb5ae4000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:15:42.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:42 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:42.480+0000 7f6eb5ae4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:15:43.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:42 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:42.593+0000 7f6eb5ae4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:15:43.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:42 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:42.678+0000 7f6eb5ae4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:15:43.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:43.216+0000 7f6eb5ae4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:15:44.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:43.570+0000 7f6eb5ae4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:15:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:15:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-24T16:15:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "initialized": true 2024-04-24T16:15:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:15:44.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:15:44] ENGINE Bus STARTING 2024-04-24T16:15:44.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:15:44] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:15:44.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:15:44] ENGINE Bus STARTED 2024-04-24T16:15:44.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: Active manager daemon y restarted 2024-04-24T16:15:44.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: Activating manager daemon y 2024-04-24T16:15:44.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: osdmap e2: 0 total, 0 up, 0 in 2024-04-24T16:15:44.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: mgrmap e6: y(active, starting, since 0.052562s) 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: Manager daemon y is now available 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:15:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:44 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:15:45.153 INFO:teuthology.orchestra.run.smithi070.stderr:mgr epoch 5 is available 2024-04-24T16:15:45.154 INFO:teuthology.orchestra.run.smithi070.stderr:Setting orchestrator backend to cephadm... 2024-04-24T16:15:46.022 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:45 smithi070 ceph-mon[21827]: [24/Apr/2024:16:15:44] ENGINE Bus STARTING 2024-04-24T16:15:46.022 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:45 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:46.022 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:45 smithi070 ceph-mon[21827]: mgrmap e7: y(active, since 1.0549s) 2024-04-24T16:15:46.022 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:45 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:15:46.675 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:46 smithi070 ceph-mon[21827]: [24/Apr/2024:16:15:44] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:15:46.676 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:46 smithi070 ceph-mon[21827]: [24/Apr/2024:16:15:44] ENGINE Bus STARTED 2024-04-24T16:15:46.676 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:46 smithi070 ceph-mon[21827]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T16:15:46.676 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:46 smithi070 ceph-mon[21827]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T16:15:46.676 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:46 smithi070 ceph-mon[21827]: mgrmap e8: y(active, since 2s) 2024-04-24T16:15:46.676 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:46 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:46.676 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:46 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:15:46.950 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: value unchanged 2024-04-24T16:15:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:Generating ssh key... 2024-04-24T16:15:47.823 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:47 smithi070 ceph-mon[21827]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:15:48.934 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-mon[21827]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: Generating public/private rsa key pair. 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: Your identification has been saved in /tmp/tmp9dvf5n8t/key. 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: Your public key has been saved in /tmp/tmp9dvf5n8t/key.pub. 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: The key fingerprint is: 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: SHA256:fIpLKCaQbtQMI0iPK0eqlCqACY1TgrTHbcpU08gA/kk ceph-be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: The key's randomart image is: 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: +---[RSA 3072]----+ 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |++o.ooo | 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |=== oo.. | 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |**oE o | 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |+*& + . | 2024-04-24T16:15:48.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |O=.B S . | 2024-04-24T16:15:48.936 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |X. . . o | 2024-04-24T16:15:48.936 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |=oo . o . | 2024-04-24T16:15:48.936 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: |oo . . . | 2024-04-24T16:15:48.936 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: | . | 2024-04-24T16:15:48.936 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: +----[SHA256]-----+ 2024-04-24T16:15:49.752 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY5J9g54hMFXtTJNAV+As7Bn7Ar5gCrgkeE5QmJ4zMjEoUcQRoxplqT/LUFcBHm3qvi6VSgfV8znOxdjcKzRN1j5rT6+8ZJ6v8Z4OVK3qUn1EYJQwUcZ1/5DoBarrUxsEDkgGVtjl3ruw2daLPmrTyJlESUYpAw6d3mrHCqHlAqsOcb2kmEJoERHWJTtDklm4EY6AoDzU1ZQy9gYy7LnjfFCA8HinyHojfHAKvottFEHxfpWJJ+EUUDa6ZNSMMJvnnQT/75uXoEckO/vSnSjHyCu3wnnouf0uRLRX+mHJRPfm1ngelWI9chYsoA1JJ519QvNSo0Pcgz9QH1O/+3sQVKOsRgOk28ejoXQYYRxLz1qPDJ7FxVlJVJBUrssptSoBaxb5XeL3ERIyrBPhMSCAsCeUb7wDrM53NZWwixWo8aMGtaNt7IDfDUVlVf+rdmvdQwHeuyehKsMkMGcfs6ecswYcj279rNFJnhzgLKp2FaSUAOrCYpkWBdZOZfYnmMpc= ceph-be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:15:50.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:49 smithi070 ceph-mon[21827]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:15:50.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:49 smithi070 ceph-mon[21827]: Generating ssh key... 2024-04-24T16:15:50.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:49 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:50.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:49 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:50.309 INFO:teuthology.orchestra.run.smithi070.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-24T16:15:50.310 INFO:teuthology.orchestra.run.smithi070.stderr:Adding key to root@localhost authorized_keys... 2024-04-24T16:15:50.310 INFO:teuthology.orchestra.run.smithi070.stderr:Adding host smithi070... 2024-04-24T16:15:50.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:50 smithi070 ceph-mon[21827]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:15:51.675 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: Added host 'smithi070' with addr '172.21.15.70' 2024-04-24T16:15:51.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:51 smithi070 ceph-mon[21827]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "addr": "172.21.15.70", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:15:51.741 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:51 smithi070 ceph-mon[21827]: Deploying cephadm binary to smithi070 2024-04-24T16:15:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:Deploying unmanaged mon service... 2024-04-24T16:15:52.894 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:52 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:52.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:52 smithi070 ceph-mon[21827]: Added host smithi070 2024-04-24T16:15:52.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:52 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:15:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-24T16:15:53.439 INFO:teuthology.orchestra.run.smithi070.stderr:Deploying unmanaged mgr service... 2024-04-24T16:15:54.193 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:53 smithi070 ceph-mon[21827]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:15:54.193 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:53 smithi070 ceph-mon[21827]: Saving service mon spec with placement count:5 2024-04-24T16:15:54.193 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:53 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:54.390 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-24T16:15:55.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:55 smithi070 ceph-mon[21827]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:15:55.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:55 smithi070 ceph-mon[21827]: Saving service mgr spec with placement count:2 2024-04-24T16:15:55.456 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:55 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:56.954 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:56 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1518978744' entity='client.admin' 2024-04-24T16:15:57.670 INFO:teuthology.orchestra.run.smithi070.stderr:Enabling the dashboard module... 2024-04-24T16:15:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:57 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/669690813' entity='client.admin' 2024-04-24T16:15:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:57 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:57 smithi070 ceph-mon[21827]: from='mgr.14120 172.21.15.70:0/118165070' entity='mgr.y' 2024-04-24T16:15:59.150 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:58 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/245147772' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-24T16:15:59.151 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setuser ceph since I am not root 2024-04-24T16:15:59.151 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setgroup ceph since I am not root 2024-04-24T16:15:59.151 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:59.024+0000 7faef050e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:15:59.151 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:59.152+0000 7faef050e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:15:59.403 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:59.251+0000 7faef050e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:15:59.671 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:59.561+0000 7faef050e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:15:59.671 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:15:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:15:59.643+0000 7faef050e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:16:00.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:59 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/245147772' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-24T16:16:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:15:59 smithi070 ceph-mon[21827]: mgrmap e9: y(active, since 15s) 2024-04-24T16:16:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:16:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-24T16:16:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "available": true, 2024-04-24T16:16:00.127 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-24T16:16:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-24T16:16:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:16:00.728 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for the mgr to restart... 2024-04-24T16:16:00.728 INFO:teuthology.orchestra.run.smithi070.stderr:Waiting for mgr epoch 9... 2024-04-24T16:16:00.982 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:00 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/660675923' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T16:16:00.982 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:00 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:00.778+0000 7faef050e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:16:00.982 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:00 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:00.945+0000 7faef050e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:16:01.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:01.032+0000 7faef050e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:16:01.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:01.347+0000 7faef050e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:16:01.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:01.435+0000 7faef050e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:16:01.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:01.526+0000 7faef050e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:16:01.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:01.761+0000 7faef050e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:16:02.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:02 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:02.196+0000 7faef050e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:16:03.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:02 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:02.671+0000 7faef050e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:16:03.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:02 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:02.764+0000 7faef050e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:16:03.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:03.053+0000 7faef050e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:16:03.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:03.141+0000 7faef050e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:16:03.877 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:03.543+0000 7faef050e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:16:03.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:03.675+0000 7faef050e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:16:03.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:03.765+0000 7faef050e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:16:03.878 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:03.876+0000 7faef050e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:16:04.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:03.964+0000 7faef050e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:16:04.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:04.500+0000 7faef050e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:16:05.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:16:04.855+0000 7faef050e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:16:05.856 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:16:05] ENGINE Bus STARTING 2024-04-24T16:16:05.856 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:16:05] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:16:05.856 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:16:05] ENGINE Bus STARTED 2024-04-24T16:16:05.918 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: { 2024-04-24T16:16:05.918 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-24T16:16:05.918 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: "initialized": true 2024-04-24T16:16:05.918 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: } 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: Active manager daemon y restarted 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: Activating manager daemon y 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: osdmap e3: 0 total, 0 up, 0 in 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: mgrmap e10: y(active, starting, since 0.0526815s) 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:16:06.219 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:16:06.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:16:06.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: Manager daemon y is now available 2024-04-24T16:16:06.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:06.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:06.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:16:06.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:16:06.220 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:06.489 INFO:teuthology.orchestra.run.smithi070.stderr:mgr epoch 9 is available 2024-04-24T16:16:06.489 INFO:teuthology.orchestra.run.smithi070.stderr:Generating a dashboard self-signed certificate... 2024-04-24T16:16:07.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:06 smithi070 ceph-mon[21827]: [24/Apr/2024:16:16:05] ENGINE Bus STARTING 2024-04-24T16:16:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:06 smithi070 ceph-mon[21827]: [24/Apr/2024:16:16:05] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:16:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:06 smithi070 ceph-mon[21827]: [24/Apr/2024:16:16:05] ENGINE Bus STARTED 2024-04-24T16:16:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:06 smithi070 ceph-mon[21827]: mgrmap e11: y(active, since 1.05516s) 2024-04-24T16:16:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:06 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:07.303 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-24T16:16:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:Creating initial admin user... 2024-04-24T16:16:08.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:07 smithi070 ceph-mon[21827]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T16:16:08.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:07 smithi070 ceph-mon[21827]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T16:16:08.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:07 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:08.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:07 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:08.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:07 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:08.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:07 smithi070 ceph-mon[21827]: mgrmap e12: y(active, since 2s) 2024-04-24T16:16:08.652 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$0.vekQlmDzYmiWjL0cpKJuklocx0irA5UE4KwjPc8dZFhZQB49TsG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713975368, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-24T16:16:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:08 smithi070 ceph-mon[21827]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:09.348 INFO:teuthology.orchestra.run.smithi070.stderr:Fetching dashboard port number... 2024-04-24T16:16:09.935 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: 8443 2024-04-24T16:16:10.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:09 smithi070 ceph-mon[21827]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:10.783 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: disabled 2024-04-24T16:16:10.789 INFO:teuthology.orchestra.run.smithi070.stderr:systemctl: inactive 2024-04-24T16:16:10.790 INFO:teuthology.orchestra.run.smithi070.stderr:firewalld.service is not enabled 2024-04-24T16:16:10.790 INFO:teuthology.orchestra.run.smithi070.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-24T16:16:10.792 INFO:teuthology.orchestra.run.smithi070.stderr:Ceph Dashboard is now available at: 2024-04-24T16:16:10.792 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:10.792 INFO:teuthology.orchestra.run.smithi070.stderr: URL: https://smithi070:8443/ 2024-04-24T16:16:10.792 INFO:teuthology.orchestra.run.smithi070.stderr: User: admin 2024-04-24T16:16:10.792 INFO:teuthology.orchestra.run.smithi070.stderr: Password: kgjj8uk4cj 2024-04-24T16:16:10.792 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:10.792 INFO:teuthology.orchestra.run.smithi070.stderr:Enabling autotune for osd_memory_target 2024-04-24T16:16:11.199 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:10 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2053703546' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-24T16:16:11.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:11 smithi070 ceph-mon[21827]: mgrmap e13: y(active, since 6s) 2024-04-24T16:16:12.782 INFO:teuthology.orchestra.run.smithi070.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr:You can access the Ceph CLI with: 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr: ceph telemetry on 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:13.346 INFO:teuthology.orchestra.run.smithi070.stderr:For more information see: 2024-04-24T16:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-24T16:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:Bootstrap complete. 2024-04-24T16:16:13.381 INFO:tasks.cephadm:Fetching config... 2024-04-24T16:16:13.382 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:16:13.382 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-24T16:16:13.397 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-24T16:16:13.398 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:16:13.398 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-24T16:16:13.452 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-24T16:16:13.453 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:16:13.453 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-24T16:16:13.517 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-24T16:16:13.517 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:16:13.517 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-24T16:16:13.572 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-24T16:16:13.572 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY5J9g54hMFXtTJNAV+As7Bn7Ar5gCrgkeE5QmJ4zMjEoUcQRoxplqT/LUFcBHm3qvi6VSgfV8znOxdjcKzRN1j5rT6+8ZJ6v8Z4OVK3qUn1EYJQwUcZ1/5DoBarrUxsEDkgGVtjl3ruw2daLPmrTyJlESUYpAw6d3mrHCqHlAqsOcb2kmEJoERHWJTtDklm4EY6AoDzU1ZQy9gYy7LnjfFCA8HinyHojfHAKvottFEHxfpWJJ+EUUDa6ZNSMMJvnnQT/75uXoEckO/vSnSjHyCu3wnnouf0uRLRX+mHJRPfm1ngelWI9chYsoA1JJ519QvNSo0Pcgz9QH1O/+3sQVKOsRgOk28ejoXQYYRxLz1qPDJ7FxVlJVJBUrssptSoBaxb5XeL3ERIyrBPhMSCAsCeUb7wDrM53NZWwixWo8aMGtaNt7IDfDUVlVf+rdmvdQwHeuyehKsMkMGcfs6ecswYcj279rNFJnhzgLKp2FaSUAOrCYpkWBdZOZfYnmMpc= ceph-be31be9a-0255-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T16:16:13.644 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY5J9g54hMFXtTJNAV+As7Bn7Ar5gCrgkeE5QmJ4zMjEoUcQRoxplqT/LUFcBHm3qvi6VSgfV8znOxdjcKzRN1j5rT6+8ZJ6v8Z4OVK3qUn1EYJQwUcZ1/5DoBarrUxsEDkgGVtjl3ruw2daLPmrTyJlESUYpAw6d3mrHCqHlAqsOcb2kmEJoERHWJTtDklm4EY6AoDzU1ZQy9gYy7LnjfFCA8HinyHojfHAKvottFEHxfpWJJ+EUUDa6ZNSMMJvnnQT/75uXoEckO/vSnSjHyCu3wnnouf0uRLRX+mHJRPfm1ngelWI9chYsoA1JJ519QvNSo0Pcgz9QH1O/+3sQVKOsRgOk28ejoXQYYRxLz1qPDJ7FxVlJVJBUrssptSoBaxb5XeL3ERIyrBPhMSCAsCeUb7wDrM53NZWwixWo8aMGtaNt7IDfDUVlVf+rdmvdQwHeuyehKsMkMGcfs6ecswYcj279rNFJnhzgLKp2FaSUAOrCYpkWBdZOZfYnmMpc= ceph-be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:13.655 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY5J9g54hMFXtTJNAV+As7Bn7Ar5gCrgkeE5QmJ4zMjEoUcQRoxplqT/LUFcBHm3qvi6VSgfV8znOxdjcKzRN1j5rT6+8ZJ6v8Z4OVK3qUn1EYJQwUcZ1/5DoBarrUxsEDkgGVtjl3ruw2daLPmrTyJlESUYpAw6d3mrHCqHlAqsOcb2kmEJoERHWJTtDklm4EY6AoDzU1ZQy9gYy7LnjfFCA8HinyHojfHAKvottFEHxfpWJJ+EUUDa6ZNSMMJvnnQT/75uXoEckO/vSnSjHyCu3wnnouf0uRLRX+mHJRPfm1ngelWI9chYsoA1JJ519QvNSo0Pcgz9QH1O/+3sQVKOsRgOk28ejoXQYYRxLz1qPDJ7FxVlJVJBUrssptSoBaxb5XeL3ERIyrBPhMSCAsCeUb7wDrM53NZWwixWo8aMGtaNt7IDfDUVlVf+rdmvdQwHeuyehKsMkMGcfs6ecswYcj279rNFJnhzgLKp2FaSUAOrCYpkWBdZOZfYnmMpc= ceph-be31be9a-0255-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T16:16:13.695 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY5J9g54hMFXtTJNAV+As7Bn7Ar5gCrgkeE5QmJ4zMjEoUcQRoxplqT/LUFcBHm3qvi6VSgfV8znOxdjcKzRN1j5rT6+8ZJ6v8Z4OVK3qUn1EYJQwUcZ1/5DoBarrUxsEDkgGVtjl3ruw2daLPmrTyJlESUYpAw6d3mrHCqHlAqsOcb2kmEJoERHWJTtDklm4EY6AoDzU1ZQy9gYy7LnjfFCA8HinyHojfHAKvottFEHxfpWJJ+EUUDa6ZNSMMJvnnQT/75uXoEckO/vSnSjHyCu3wnnouf0uRLRX+mHJRPfm1ngelWI9chYsoA1JJ519QvNSo0Pcgz9QH1O/+3sQVKOsRgOk28ejoXQYYRxLz1qPDJ7FxVlJVJBUrssptSoBaxb5XeL3ERIyrBPhMSCAsCeUb7wDrM53NZWwixWo8aMGtaNt7IDfDUVlVf+rdmvdQwHeuyehKsMkMGcfs6ecswYcj279rNFJnhzgLKp2FaSUAOrCYpkWBdZOZfYnmMpc= ceph-be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:13.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-24T16:16:14.035 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:13 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1417447853' entity='client.admin' 2024-04-24T16:16:16.468 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:16.468 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:16:16.468 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:16.468 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:16.469 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:16.734 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-24T16:16:16.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-24T16:16:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:17 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3488456050' entity='client.admin' 2024-04-24T16:16:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:17 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:17 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:19.162 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2024-04-24T16:16:19.162 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T16:16:19.162 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2024-04-24T16:16:19.180 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T16:16:19.181 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:19.237 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2024-04-24T16:16:19.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch host add smithi156 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:21.846 INFO:teuthology.orchestra.run.smithi070.stdout:Added host 'smithi156' with addr '172.21.15.156' 2024-04-24T16:16:22.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: Deploying cephadm binary to smithi156 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: Added host smithi156 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:23.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:24.343 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:16:24.344 INFO:teuthology.orchestra.run.smithi070.stdout:[{"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2024-04-24T16:16:24.948 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:24 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:24.973 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-24T16:16:24.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-24T16:16:25.914 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:25 smithi070 ceph-mon[21827]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T16:16:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:26 smithi070 ceph-mon[21827]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:16:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-04-24T16:16:27.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:27 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/243882550' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T16:16:28.225 INFO:tasks.cephadm:Adding mon.a on smithi070 2024-04-24T16:16:28.225 INFO:tasks.cephadm:Adding mon.c on smithi070 2024-04-24T16:16:28.226 INFO:tasks.cephadm:Adding mon.b on smithi156 2024-04-24T16:16:28.226 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b' 2024-04-24T16:16:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:28 smithi070 ceph-mon[21827]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:28 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/243882550' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T16:16:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:28 smithi070 ceph-mon[21827]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:30.279 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mon update... 2024-04-24T16:16:31.027 DEBUG:teuthology.orchestra.run.smithi070:mon.c> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.c.service 2024-04-24T16:16:31.030 DEBUG:teuthology.orchestra.run.smithi156:mon.b> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.b.service 2024-04-24T16:16:31.032 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T16:16:31.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T16:16:31.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:31.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:31.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:31.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:31.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:32.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:31 smithi070 ceph-mon[21827]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:32.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:31 smithi070 ceph-mon[21827]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b;count:3 2024-04-24T16:16:32.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:31 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:16:32.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:31 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:32.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:31 smithi070 ceph-mon[21827]: Deploying daemon mon.c on smithi070 2024-04-24T16:16:32.768 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:16:32.768 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":1,"fsid":"be31be9a-0255-11ef-bc93-c7b262605968","modified":"2024-04-24T16:15:12.900488Z","created":"2024-04-24T16:15:12.900488Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-24T16:16:32.770 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 1 2024-04-24T16:16:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:32 smithi070 ceph-mon[21827]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:33.439 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 systemd[1]: Starting Ceph mon.c for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:16:33.689 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/975980482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:33.690 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 podman[26274]: 2024-04-24 16:16:33.442000815 +0000 UTC m=+0.099008360 container create 2b11cd568406b08c479d0dac41dda90a05602e5bc3e9b024e45cb0cba1c33d96 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, version=8, vendor=Red Hat, Inc., vcs-type=git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , release=754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-24T16:16:33.690 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 podman[26274]: 2024-04-24 16:16:33.361851039 +0000 UTC m=+0.018858591 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-24T16:16:33.690 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 podman[26274]: 2024-04-24 16:16:33.644278642 +0000 UTC m=+0.301286185 container init 2b11cd568406b08c479d0dac41dda90a05602e5bc3e9b024e45cb0cba1c33d96 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 podman[26274]: 2024-04-24 16:16:33.648186013 +0000 UTC m=+0.305193556 container start 2b11cd568406b08c479d0dac41dda90a05602e5bc3e9b024e45cb0cba1c33d96 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, name=centos-stream, GIT_CLEAN=True, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, ceph=True, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: pidfile_write: ignore empty --pid-file 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: load: jerasure load: lrc 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: RocksDB version: 6.15.5 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Compile date Apr 18 2022 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: DB SUMMARY 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: DB Session ID: 52OZHQ2D6DK6O3OJQ0P6 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: CURRENT file: CURRENT 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T16:16:33.691 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.error_if_exists: 0 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.create_if_missing: 0 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.paranoid_checks: 1 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.env: 0x55ffa42c6860 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.fs: Posix File System 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.info_log: 0x55ffa64f3dc0 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.statistics: (nil) 2024-04-24T16:16:33.692 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.use_fsync: 0 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_log_file_size: 0 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.allow_fallocate: 1 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T16:16:33.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.use_direct_reads: 0 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.db_log_dir: 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T16:16:33.695 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.write_buffer_manager: 0x55ffa65e42d0 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T16:16:33.696 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.unordered_write: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.row_cache: None 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.wal_filter: None 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.preserve_deletes: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.two_write_queues: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.atomic_flush: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T16:16:33.697 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.log_readahead_size: 0 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_background_jobs: 2 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_background_compactions: -1 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_subcompactions: 1 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T16:16:33.698 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_open_files: -1 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_background_flushes: -1 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Compression algorithms supported: 2024-04-24T16:16:33.699 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kZSTD supported: 0 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kXpressCompression supported: 0 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kLZ4Compression supported: 1 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kBZip2Compression supported: 0 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kZlibCompression supported: 1 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: kSnappyCompression supported: 1 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.merge_operator: 2024-04-24T16:16:33.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_filter: None 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ffa64bfd00) 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_type: 0 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_index_type: 0 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_shortening: 1 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: hash_index_allow_collision: 1 2024-04-24T16:16:33.701 INFO:journalctl@ceph.mon.c.smithi070.stdout: checksum: 1 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: no_block_cache: 0 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache: 0x55ffa652a170 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_options: 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: capacity : 536870912 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: num_shard_bits : 4 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: strict_capacity_limit : 0 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_compressed: (nil) 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: persistent_cache: (nil) 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size: 4096 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size_deviation: 10 2024-04-24T16:16:33.702 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_restart_interval: 16 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_block_restart_interval: 1 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: metadata_block_size: 4096 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: partition_filters: 0 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_delta_encoding: 1 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: whole_key_filtering: 1 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: verify_compression: 0 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: format_version: 4 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_index_compression: 1 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_align: 0 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression: NoCompression 2024-04-24T16:16:33.703 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.num_levels: 7 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T16:16:33.704 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T16:16:33.705 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.arena_block_size: 4194304 2024-04-24T16:16:33.706 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T16:16:33.707 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.table_properties_collectors: 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.inplace_update_support: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.bloom_locality: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.max_successive_merges: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.ttl: 2592000 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.enable_blob_files: false 2024-04-24T16:16:33.708 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.min_blob_size: 0 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975393674845, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975393675265, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713975393, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "2a829568-bc51-4ade-8565-e10bd2a09140", "db_session_id": "52OZHQ2D6DK6O3OJQ0P6"}} 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975393675674, "job": 1, "event": "recovery_finished"} 2024-04-24T16:16:33.709 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55ffa6510a80 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: DB pointer 0x55ffa6520000 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** DB Stats ** 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-24T16:16:33.710 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:16:33.711 INFO:journalctl@ceph.mon.c.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:33.712 INFO:journalctl@ceph.mon.c.smithi070.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:33.713 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: using public_addrv [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: starting mon.c rank -1 at public addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] at bind addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(???) e0 preinit fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).mds e1 new map 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout: e1 2024-04-24T16:16:33.714 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout: legacy client fscid: -1 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout: No filesystems configured 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T16:16:33.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b;count:3 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: Deploying daemon mon.c on smithi070 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/975980482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-24T16:16:33.716 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 ceph-mon[26293]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-24T16:16:34.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 bash[26274]: 2b11cd568406b08c479d0dac41dda90a05602e5bc3e9b024e45cb0cba1c33d96 2024-04-24T16:16:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:33 smithi070 systemd[1]: Started Ceph mon.c for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:16:34.331 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T16:16:34.332 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: mon.a calling monitor election 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: mon.c calling monitor election 2024-04-24T16:16:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: monmap e2: 2 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: fsmap 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: mgrmap e13: y(active, since 33s) 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: overall HEALTH_OK 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:39.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: mon.a calling monitor election 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: mon.c calling monitor election 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: monmap e2: 2 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: fsmap 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: mgrmap e13: y(active, since 33s) 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: overall HEALTH_OK 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:39.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:39.297 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:16:39.297 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"fsid":"be31be9a-0255-11ef-bc93-c7b262605968","modified":"2024-04-24T16:16:33.695018Z","created":"2024-04-24T16:15:12.900488Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3301","nonce":0},{"type":"v1","addr":"172.21.15.70:6790","nonce":0}]},"addr":"172.21.15.70:6790/0","public_addr":"172.21.15.70:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T16:16:39.300 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 2 2024-04-24T16:16:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:39 smithi070 ceph-mon[21827]: Deploying daemon mon.b on smithi156 2024-04-24T16:16:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:39 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/1370013319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:39 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:40.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:39 smithi070 ceph-mon[26293]: Deploying daemon mon.b on smithi156 2024-04-24T16:16:40.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:39 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/1370013319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:40.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:39 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:41.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:40 smithi070 ceph-mon[21827]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:41.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:40 smithi070 ceph-mon[26293]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:41.229 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T16:16:41.229 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T16:16:42.553 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:42 smithi156 systemd[1]: Starting Ceph mon.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:16:43.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:42 smithi070 ceph-mon[21827]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:43.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:42 smithi070 ceph-mon[26293]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:43.461 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:16:43.461 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"fsid":"be31be9a-0255-11ef-bc93-c7b262605968","modified":"2024-04-24T16:16:33.695018Z","created":"2024-04-24T16:15:12.900488Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3301","nonce":0},{"type":"v1","addr":"172.21.15.70:6790","nonce":0}]},"addr":"172.21.15.70:6790/0","public_addr":"172.21.15.70:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T16:16:43.461 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:43 smithi156 podman[22644]: 2024-04-24 16:16:43.247119655 +0000 UTC m=+0.022971103 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-24T16:16:43.462 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 2 2024-04-24T16:16:43.831 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:43 smithi156 podman[22644]: 2024-04-24 16:16:43.488404621 +0000 UTC m=+0.264256069 container create 9f6c28bccd1d0d19347638e046f9735cf19ef608a9eab9be58a694ed98ba8cf4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, ceph=True, version=8, RELEASE=HEAD, release=754, vcs-type=git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-24T16:16:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:43 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/1493702932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:43 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/1493702932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:44.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 podman[22644]: 2024-04-24 16:16:44.222572175 +0000 UTC m=+0.998423623 container init 9f6c28bccd1d0d19347638e046f9735cf19ef608a9eab9be58a694ed98ba8cf4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., vcs-type=git, io.buildah.version=1.19.8, ceph=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, distribution-scope=public, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, maintainer=Guillaume Abrioux , release=754) 2024-04-24T16:16:44.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 podman[22644]: 2024-04-24 16:16:44.225864138 +0000 UTC m=+1.001715586 container start 9f6c28bccd1d0d19347638e046f9735cf19ef608a9eab9be58a694ed98ba8cf4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-17.2.0, version=8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.component=centos-stream-container, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.buildah.version=1.19.8) 2024-04-24T16:16:44.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T16:16:44.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-24T16:16:44.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: pidfile_write: ignore empty --pid-file 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: load: jerasure load: lrc 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: RocksDB version: 6.15.5 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Compile date Apr 18 2022 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: DB SUMMARY 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: DB Session ID: G95IV7NWU2MKBASADTXJ 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: CURRENT file: CURRENT 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.error_if_exists: 0 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.create_if_missing: 0 2024-04-24T16:16:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.paranoid_checks: 1 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.env: 0x5568440c6860 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.fs: Posix File System 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.info_log: 0x556844ddfdc0 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.statistics: (nil) 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.use_fsync: 0 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_log_file_size: 0 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T16:16:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.allow_fallocate: 1 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.use_direct_reads: 0 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.db_log_dir: 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T16:16:44.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.write_buffer_manager: 0x556844ed0240 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T16:16:44.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.unordered_write: 0 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.row_cache: None 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.wal_filter: None 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.preserve_deletes: 0 2024-04-24T16:16:44.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.two_write_queues: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.atomic_flush: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.log_readahead_size: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_background_jobs: 2 2024-04-24T16:16:44.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_background_compactions: -1 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_subcompactions: 1 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_open_files: -1 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T16:16:44.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_background_flushes: -1 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Compression algorithms supported: 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kZSTD supported: 0 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kXpressCompression supported: 0 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kLZ4Compression supported: 1 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kBZip2Compression supported: 0 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kZlibCompression supported: 1 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: kSnappyCompression supported: 1 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T16:16:44.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.merge_operator: 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_filter: None 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556844dabd00) 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks: 1 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T16:16:44.418 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_top_level_index_and_filter: 1 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_type: 0 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_index_type: 0 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_shortening: 1 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: hash_index_allow_collision: 1 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: checksum: 1 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: no_block_cache: 0 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache: 0x556844e16170 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_name: BinnedLRUCache 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_options: 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: capacity : 536870912 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: num_shard_bits : 4 2024-04-24T16:16:44.419 INFO:journalctl@ceph.mon.b.smithi156.stdout: strict_capacity_limit : 0 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: high_pri_pool_ratio: 0.000 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_compressed: (nil) 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: persistent_cache: (nil) 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size: 4096 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size_deviation: 10 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_restart_interval: 16 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_block_restart_interval: 1 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: metadata_block_size: 4096 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: partition_filters: 0 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_delta_encoding: 1 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: whole_key_filtering: 1 2024-04-24T16:16:44.420 INFO:journalctl@ceph.mon.b.smithi156.stdout: verify_compression: 0 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout: read_amp_bytes_per_bit: 0 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout: format_version: 4 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_index_compression: 1 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_align: 0 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression: NoCompression 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.num_levels: 7 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T16:16:44.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:16:44.422 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T16:16:44.423 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.arena_block_size: 4194304 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T16:16:44.424 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.table_properties_collectors: 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.inplace_update_support: 0 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.bloom_locality: 0 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.max_successive_merges: 0 2024-04-24T16:16:44.425 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.ttl: 2592000 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.enable_blob_files: false 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.min_blob_size: 0 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T16:16:44.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975404252296, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975404252598, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713975404, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "d674013e-90cc-4f1c-a0bd-31520bd90ae9", "db_session_id": "G95IV7NWU2MKBASADTXJ"}} 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713975404252935, "job": 1, "event": "recovery_finished"} 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x556844dfca80 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: DB pointer 0x556844e0c000 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-24T16:16:44.427 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** DB Stats ** 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:44.428 INFO:journalctl@ceph.mon.b.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:44.429 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-24T16:16:44.430 INFO:journalctl@ceph.mon.b.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-24T16:16:44.431 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: starting mon.b rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(???) e0 preinit fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).mds e1 new map 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout: e1 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout: legacy client fscid: -1 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout: No filesystems configured 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-24T16:16:44.432 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b;count:3 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: Deploying daemon mon.c on smithi070 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:44.433 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/975980482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.a calling monitor election 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.c calling monitor election 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:44.434 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: monmap e2: 2 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: fsmap 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mgrmap e13: y(active, since 33s) 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: overall HEALTH_OK 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: Deploying daemon mon.b on smithi156 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/1370013319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:44.435 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:44.436 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/1493702932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:44.436 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-04-24T16:16:44.436 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 ceph-mon[22679]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-24T16:16:44.436 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 bash[22644]: 9f6c28bccd1d0d19347638e046f9735cf19ef608a9eab9be58a694ed98ba8cf4 2024-04-24T16:16:44.436 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:44 smithi156 systemd[1]: Started Ceph mon.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:16:45.080 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T16:16:45.080 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T16:16:49.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:16:49.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: mon.c calling monitor election 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: mon.a calling monitor election 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: fsmap 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: mgrmap e13: y(active, since 44s) 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: mon.b (rank 2) addr [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] is down (out of quorum) 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:16:49.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: mon.c calling monitor election 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: mon.a calling monitor election 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: fsmap 2024-04-24T16:16:49.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: mgrmap e13: y(active, since 44s) 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: mon.b (rank 2) addr [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] is down (out of quorum) 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:49.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:49 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:49.814 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:16:49.814 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"fsid":"be31be9a-0255-11ef-bc93-c7b262605968","modified":"2024-04-24T16:16:44.272729Z","created":"2024-04-24T16:15:12.900488Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3301","nonce":0},{"type":"v1","addr":"172.21.15.70:6790","nonce":0}]},"addr":"172.21.15.70:6790/0","public_addr":"172.21.15.70:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T16:16:49.816 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 3 2024-04-24T16:16:50.434 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-24T16:16:50.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-24T16:16:50.678 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[26293]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/1597951368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[21827]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/1597951368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:50.679 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:51.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:16:51.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.c calling monitor election 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.a calling monitor election 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:51.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: fsmap 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mgrmap e13: y(active, since 44s) 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.b (rank 2) addr [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] is down (out of quorum) 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/1597951368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:51.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:51.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:52.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: mon.b calling monitor election 2024-04-24T16:16:52.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:52.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T16:16:52.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: Reconfiguring daemon mon.a on smithi070 2024-04-24T16:16:52.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: mon.b calling monitor election 2024-04-24T16:16:52.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: mon.c calling monitor election 2024-04-24T16:16:52.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: mon.a calling monitor election 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: fsmap 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: mgrmap e13: y(active, since 46s) 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: Cluster is now healthy 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[21827]: overall HEALTH_OK 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: mon.b calling monitor election 2024-04-24T16:16:52.218 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: Reconfiguring daemon mon.a on smithi070 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: mon.b calling monitor election 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: mon.c calling monitor election 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: mon.a calling monitor election 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: fsmap 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: mgrmap e13: y(active, since 46s) 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: Cluster is now healthy 2024-04-24T16:16:52.219 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:51 smithi070 ceph-mon[26293]: overall HEALTH_OK 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.b calling monitor election 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: Reconfiguring daemon mon.a on smithi070 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.b calling monitor election 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.c calling monitor election 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.a calling monitor election 2024-04-24T16:16:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:16:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:16:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: fsmap 2024-04-24T16:16:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T16:16:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: mgrmap e13: y(active, since 46s) 2024-04-24T16:16:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-24T16:16:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: Cluster is now healthy 2024-04-24T16:16:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:51 smithi156 ceph-mon[22679]: overall HEALTH_OK 2024-04-24T16:16:52.797 INFO:teuthology.orchestra.run.smithi070.stdout:# minimal ceph.conf for be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:52.897 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-04-24T16:16:52.898 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:16:52.898 INFO:teuthology.orchestra.run.smithi070.stdout: mon_host = [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] 2024-04-24T16:16:53.097 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:52 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:53.097 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:52 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/278457656' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:53.098 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:52 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:53.098 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:52 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/278457656' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:53.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:52 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:16:53.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:52 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/278457656' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:53.743 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-24T16:16:53.743 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:16:53.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T16:16:53.768 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:16:53.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:53.835 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T16:16:53.835 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T16:16:53.865 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T16:16:53.865 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:16:53.932 INFO:tasks.cephadm:Adding mgr.y on smithi070 2024-04-24T16:16:53.933 INFO:tasks.cephadm:Adding mgr.x on smithi156 2024-04-24T16:16:53.933 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi070=y;smithi156=x' 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[21827]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[21827]: Reconfiguring mon.c (monmap changed)... 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[21827]: Reconfiguring daemon mon.c on smithi070 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[26293]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[26293]: Reconfiguring mon.c (monmap changed)... 2024-04-24T16:16:54.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:54.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:16:54.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:54.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:54 smithi070 ceph-mon[26293]: Reconfiguring daemon mon.c on smithi070 2024-04-24T16:16:54.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:54 smithi156 ceph-mon[22679]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:54.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:54 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:54.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:54 smithi156 ceph-mon[22679]: Reconfiguring mon.c (monmap changed)... 2024-04-24T16:16:54.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:54 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:54.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:54 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:16:54.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:54 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:54.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:54 smithi156 ceph-mon[22679]: Reconfiguring daemon mon.c on smithi070 2024-04-24T16:16:55.825 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mgr update... 2024-04-24T16:16:56.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[21827]: Reconfiguring mon.b (monmap changed)... 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[21827]: Reconfiguring daemon mon.b on smithi156 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[26293]: Reconfiguring mon.b (monmap changed)... 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:56.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:16:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:55 smithi070 ceph-mon[26293]: Reconfiguring daemon mon.b on smithi156 2024-04-24T16:16:56.134 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:56.134 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:55 smithi156 ceph-mon[22679]: Reconfiguring mon.b (monmap changed)... 2024-04-24T16:16:56.134 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:16:56.134 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:16:56.135 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:56.135 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:55 smithi156 ceph-mon[22679]: Reconfiguring daemon mon.b on smithi156 2024-04-24T16:16:56.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:56 smithi156 ceph-mon[22679]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:56.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:56 smithi156 ceph-mon[22679]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi070=y;smithi156=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:56.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:56 smithi156 ceph-mon[22679]: Saving service mgr spec with placement smithi070=y;smithi156=x;count:2 2024-04-24T16:16:56.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:56 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:56.836 DEBUG:teuthology.orchestra.run.smithi156:mgr.x> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service 2024-04-24T16:16:56.838 INFO:tasks.cephadm:Deploying OSDs... 2024-04-24T16:16:56.839 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:16:56.839 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T16:16:56.858 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T16:16:56.858 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-04-24T16:16:56.915 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T16:16:56.915 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-24T16:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-24 16:16:33.188984034 +0000 2024-04-24T16:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-24 16:14:17.955254291 +0000 2024-04-24T16:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-24 16:14:17.955254291 +0000 2024-04-24T16:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-24 16:14:17.955254291 +0000 2024-04-24T16:16:56.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T16:16:56.983 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-24T16:16:56.983 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-24T16:16:56.983 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000100214 s, 5.1 MB/s 2024-04-24T16:16:56.985 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T16:16:57.043 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-24 16:16:33.188984034 +0000 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-24 16:14:18.280244546 +0000 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-24 16:14:18.280244546 +0000 2024-04-24T16:16:57.101 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-24 16:14:18.280244546 +0000 2024-04-24T16:16:57.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T16:16:57.164 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[26293]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:57.164 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[26293]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi070=y;smithi156=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:57.164 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[26293]: Saving service mgr spec with placement smithi070=y;smithi156=x;count:2 2024-04-24T16:16:57.164 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:57.164 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[21827]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:57.164 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[21827]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi070=y;smithi156=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:16:57.165 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[21827]: Saving service mgr spec with placement smithi070=y;smithi156=x;count:2 2024-04-24T16:16:57.165 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:56 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-24T16:16:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-24T16:16:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000117178 s, 4.4 MB/s 2024-04-24T16:16:57.167 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T16:16:57.225 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-24 16:16:33.189984005 +0000 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-24 16:14:18.624234231 +0000 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-24 16:14:18.624234231 +0000 2024-04-24T16:16:57.282 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-24 16:14:18.624234231 +0000 2024-04-24T16:16:57.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T16:16:57.345 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-24T16:16:57.345 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-24T16:16:57.345 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000116376 s, 4.4 MB/s 2024-04-24T16:16:57.346 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T16:16:57.403 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-24 16:16:33.189984005 +0000 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-24 16:14:18.960224156 +0000 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-24 16:14:18.960224156 +0000 2024-04-24T16:16:57.458 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-24 16:14:18.960224156 +0000 2024-04-24T16:16:57.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T16:16:57.520 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-24T16:16:57.520 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-24T16:16:57.520 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000112544 s, 4.5 MB/s 2024-04-24T16:16:57.521 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T16:16:57.577 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T16:16:57.578 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T16:16:57.593 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T16:16:57.593 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 16:16:42.417542920 +0000 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 16:14:14.727198749 +0000 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 16:14:14.727198749 +0000 2024-04-24T16:16:57.650 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 16:14:14.727198749 +0000 2024-04-24T16:16:57.651 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T16:16:57.714 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T16:16:57.714 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T16:16:57.714 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000109394 s, 4.7 MB/s 2024-04-24T16:16:57.715 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T16:16:57.771 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 16:16:42.417542920 +0000 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 16:14:15.206191290 +0000 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 16:14:15.206191290 +0000 2024-04-24T16:16:57.827 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 16:14:15.206191290 +0000 2024-04-24T16:16:57.829 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T16:16:57.889 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T16:16:57.889 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T16:16:57.889 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.5775e-05 s, 6.0 MB/s 2024-04-24T16:16:57.890 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T16:16:57.949 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-04-24T16:16:58.007 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T16:16:58.007 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:58.007 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T16:16:58.007 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:58.007 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:58.007 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 16:16:42.417542920 +0000 2024-04-24T16:16:58.008 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 16:14:15.527186291 +0000 2024-04-24T16:16:58.008 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 16:14:15.527186291 +0000 2024-04-24T16:16:58.008 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 16:14:15.527186291 +0000 2024-04-24T16:16:58.008 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T16:16:58.076 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T16:16:58.076 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T16:16:58.076 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000115858 s, 4.4 MB/s 2024-04-24T16:16:58.077 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T16:16:58.135 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-04-24T16:16:58.191 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 16:16:42.418542905 +0000 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 16:14:15.859181121 +0000 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 16:14:15.859181121 +0000 2024-04-24T16:16:58.192 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 16:14:15.859181121 +0000 2024-04-24T16:16:58.192 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T16:16:58.254 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T16:16:58.254 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T16:16:58.255 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000109949 s, 4.7 MB/s 2024-04-24T16:16:58.256 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T16:16:58.312 INFO:tasks.cephadm:Deploying osd.0 on smithi070 with /dev/vg_nvme/lv_4... 2024-04-24T16:16:58.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-04-24T16:16:58.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:58.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[21827]: Deploying daemon mgr.x on smithi156 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:16:58 smithi070 ceph-mon[26293]: Deploying daemon mgr.x on smithi156 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:16:58.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:16:58 smithi156 ceph-mon[22679]: Deploying daemon mgr.x on smithi156 2024-04-24T16:16:59.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:16:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:16:59.741+0000 7f5570954000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:17:00.408 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:00.065+0000 7f5570954000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:17:00.408 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:00.159+0000 7f5570954000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:17:00.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[26293]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:00.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:00.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:00.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:00.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:00.646 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:00.647 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[21827]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:00.647 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:00.647 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:00.647 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:00.647 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:00.647 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:00.893 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:17:00.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-mon[22679]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:00.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:00.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:00.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:00.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:00.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:00.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2024-04-24T16:17:01.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:01.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:01.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-mon[22679]: Reconfiguring mgr.y (unknown last config time)... 2024-04-24T16:17:01.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:17:01.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:17:01.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:01.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-mon[22679]: Reconfiguring daemon mgr.y on smithi070 2024-04-24T16:17:01.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:01.298+0000 7f5570954000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:17:01.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:01.461+0000 7f5570954000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:17:01.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:01.543+0000 7f5570954000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:17:01.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:01.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:01.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[21827]: Reconfiguring mgr.y (unknown last config time)... 2024-04-24T16:17:01.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:17:01.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[21827]: Reconfiguring daemon mgr.y on smithi070 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[26293]: Reconfiguring mgr.y (unknown last config time)... 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:01.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:01 smithi070 ceph-mon[26293]: Reconfiguring daemon mgr.y on smithi070 2024-04-24T16:17:02.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:01.854+0000 7f5570954000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:17:02.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:01.944+0000 7f5570954000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:17:02.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:02.036+0000 7f5570954000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:17:02.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[26293]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[21827]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:02.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:02.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:02.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:02.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:02.278+0000 7f5570954000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:17:02.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-mon[22679]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:02.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:02.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:02.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:02.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:02.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:03.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:02.710+0000 7f5570954000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:17:03.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:03.194+0000 7f5570954000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:17:03.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:03.286+0000 7f5570954000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:17:03.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:03 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:03.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:03 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:03.566 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:03 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:03.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:03 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:03.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:03 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:03.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:03 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:03.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:03 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:03.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:03 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:03.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:03 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:03.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:03.575+0000 7f5570954000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:17:03.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:03.663+0000 7f5570954000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:17:04.397 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:04.064+0000 7f5570954000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:17:04.397 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:04.195+0000 7f5570954000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:17:04.397 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:04.286+0000 7f5570954000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:17:04.397 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:04.399+0000 7f5570954000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:17:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:04 smithi156 ceph-mon[22679]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:04 smithi156 ceph-mon[22679]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:04.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:04.487+0000 7f5570954000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:17:04.802 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:04 smithi070 ceph-mon[21827]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:04.802 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:04 smithi070 ceph-mon[21827]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:04.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:04 smithi070 ceph-mon[26293]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:04.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:04 smithi070 ceph-mon[26293]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:05.390 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:05.025+0000 7f5570954000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:17:05.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:17:05.392+0000 7f5570954000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:17:05.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:05.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:17:05.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:17:05.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:17:05.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:17:05.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:17:05.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:17:05.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:05 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:17:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:17:06.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:05 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/421194417' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:17:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:06 smithi156 ceph-mon[22679]: mgrmap e14: y(active, since 60s), standbys: x 2024-04-24T16:17:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:06 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:17:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:06 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3305949732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1479dc88-7964-4663-a0e4-7c385281030b"}]: dispatch 2024-04-24T16:17:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:06 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3305949732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1479dc88-7964-4663-a0e4-7c385281030b"}]': finished 2024-04-24T16:17:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:06 smithi156 ceph-mon[22679]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T16:17:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:06 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:06 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2322950707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:07.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[21827]: mgrmap e14: y(active, since 60s), standbys: x 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3305949732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1479dc88-7964-4663-a0e4-7c385281030b"}]: dispatch 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3305949732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1479dc88-7964-4663-a0e4-7c385281030b"}]': finished 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[21827]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2322950707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[26293]: mgrmap e14: y(active, since 60s), standbys: x 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3305949732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1479dc88-7964-4663-a0e4-7c385281030b"}]: dispatch 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3305949732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1479dc88-7964-4663-a0e4-7c385281030b"}]': finished 2024-04-24T16:17:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[26293]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T16:17:07.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:07.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:06 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2322950707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:07.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:07 smithi156 ceph-mon[22679]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:08.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:07 smithi070 ceph-mon[21827]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:08.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:07 smithi070 ceph-mon[26293]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:10.039 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:09 smithi070 ceph-mon[21827]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:10.039 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:09 smithi070 ceph-mon[26293]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:10.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:09 smithi156 ceph-mon[22679]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:11.021 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:10 smithi070 ceph-mon[21827]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:11.021 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:10 smithi070 ceph-mon[26293]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:11.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:10 smithi156 ceph-mon[22679]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:12.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:11 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:17:12.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:11 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:11 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:17:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:11 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:12.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:11 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:17:12.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:11 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:13 smithi070 ceph-mon[21827]: Deploying daemon osd.0 on smithi070 2024-04-24T16:17:13.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:13 smithi070 ceph-mon[26293]: Deploying daemon osd.0 on smithi070 2024-04-24T16:17:13.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:13 smithi156 ceph-mon[22679]: Deploying daemon osd.0 on smithi070 2024-04-24T16:17:14.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:14 smithi156 ceph-mon[22679]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:14.755 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:14 smithi070 ceph-mon[26293]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:14.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:14 smithi070 ceph-mon[21827]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:16.780 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:16 smithi070 ceph-mon[21827]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:16.781 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:16 smithi070 ceph-mon[26293]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:16.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:16 smithi156 ceph-mon[22679]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:18.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[21827]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:17 smithi070 ceph-mon[26293]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:17 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:17 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:17 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:17 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:17 smithi156 ceph-mon[22679]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:19.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:18 smithi070 ceph-mon[26293]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:19.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:18 smithi070 ceph-mon[21827]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:19.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:18 smithi156 ceph-mon[22679]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:19.710 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 0 on host 'smithi070' 2024-04-24T16:17:20.308 DEBUG:teuthology.orchestra.run.smithi070:osd.0> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.0.service 2024-04-24T16:17:20.311 INFO:tasks.cephadm:Deploying osd.1 on smithi070 with /dev/vg_nvme/lv_3... 2024-04-24T16:17:20.311 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-04-24T16:17:20.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:20.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[21827]: from='osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:17:20.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[21827]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:17:20.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:20.964 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:20.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[26293]: from='osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:17:20.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[26293]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:17:20.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:20 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:21.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:20 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:21.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:20 smithi156 ceph-mon[22679]: from='osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:17:21.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:20 smithi156 ceph-mon[22679]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:17:21.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:20 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:21.971 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[21827]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T16:17:21.971 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[21827]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T16:17:21.971 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:21.971 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[21827]: from='osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[21827]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[21827]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[26293]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[26293]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[26293]: from='osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[26293]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:21.972 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-mon[26293]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:21.973 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[29173]: 2024-04-24T16:17:21.721+0000 7f9b88a0b700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T16:17:21.973 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:17:21 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[29173]: 2024-04-24T16:17:21.723+0000 7f9b8139e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:17:22.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:21 smithi156 ceph-mon[22679]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T16:17:22.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:21 smithi156 ceph-mon[22679]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T16:17:22.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:21 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:22.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:21 smithi156 ceph-mon[22679]: from='osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:22.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:21 smithi156 ceph-mon[22679]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:22.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:21 smithi156 ceph-mon[22679]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:22.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[21827]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:17:22.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[21827]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T16:17:22.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:22.853 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:22.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[26293]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:17:22.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[26293]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T16:17:22.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:22.854 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:22 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:22 smithi156 ceph-mon[22679]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:17:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:22 smithi156 ceph-mon[22679]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T16:17:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:22 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:22 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:23.721 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:17:23.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2024-04-24T16:17:23.994 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:17:23.994 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:17:23.994 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[26293]: osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718] boot 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[26293]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[26293]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[21827]: osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718] boot 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[21827]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:23.995 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:23 smithi070 ceph-mon[21827]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:23 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:17:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:23 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:17:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:23 smithi156 ceph-mon[22679]: osd.0 [v2:172.21.15.70:6802/351334718,v1:172.21.15.70:6803/351334718] boot 2024-04-24T16:17:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:23 smithi156 ceph-mon[22679]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T16:17:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:23 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:17:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:23 smithi156 ceph-mon[22679]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T16:17:24.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:24 smithi156 ceph-mon[22679]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T16:17:24.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:24 smithi070 ceph-mon[21827]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T16:17:24.926 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:24 smithi070 ceph-mon[26293]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T16:17:25.935 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:25 smithi070 ceph-mon[21827]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:25.935 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:25 smithi070 ceph-mon[26293]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:25 smithi156 ceph-mon[22679]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:27.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:27.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:27.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:26 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:26 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:27.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:27 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi070 2024-04-24T16:17:27.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:27 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi070 to 16064M 2024-04-24T16:17:27.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:27 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi070 2024-04-24T16:17:27.768 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:27 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi070 to 16064M 2024-04-24T16:17:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:27 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi070 2024-04-24T16:17:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:27 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi070 to 16064M 2024-04-24T16:17:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[21827]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1180723414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]: dispatch 2024-04-24T16:17:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]: dispatch 2024-04-24T16:17:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]': finished 2024-04-24T16:17:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[21827]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T16:17:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:29.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[26293]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:29.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1180723414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]: dispatch 2024-04-24T16:17:29.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]: dispatch 2024-04-24T16:17:29.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]': finished 2024-04-24T16:17:29.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[26293]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T16:17:29.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:28 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:28 smithi156 ceph-mon[22679]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:28 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1180723414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]: dispatch 2024-04-24T16:17:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:28 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]: dispatch 2024-04-24T16:17:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:28 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48295f9b-7ffc-4b68-99b6-7599af8a0aff"}]': finished 2024-04-24T16:17:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:28 smithi156 ceph-mon[22679]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T16:17:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:28 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:30.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:29 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2793055192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:30.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:29 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2793055192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:29 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2793055192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:31.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:30 smithi070 ceph-mon[26293]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:30 smithi070 ceph-mon[21827]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:31.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:30 smithi156 ceph-mon[22679]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:32 smithi070 ceph-mon[21827]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:32 smithi070 ceph-mon[26293]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:33.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:32 smithi156 ceph-mon[22679]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:35.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:34 smithi070 ceph-mon[21827]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:35.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:17:35.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:35.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:34 smithi070 ceph-mon[26293]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:35.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:17:35.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:34 smithi156 ceph-mon[22679]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:17:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:36.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:35 smithi070 ceph-mon[21827]: Deploying daemon osd.1 on smithi070 2024-04-24T16:17:36.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:35 smithi070 ceph-mon[26293]: Deploying daemon osd.1 on smithi070 2024-04-24T16:17:36.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:35 smithi156 ceph-mon[22679]: Deploying daemon osd.1 on smithi070 2024-04-24T16:17:37.017 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:36 smithi070 ceph-mon[21827]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:37.017 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:36 smithi070 ceph-mon[26293]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:36 smithi156 ceph-mon[22679]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:39.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:38 smithi070 ceph-mon[26293]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:38 smithi070 ceph-mon[21827]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:39.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:38 smithi156 ceph-mon[22679]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:40.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[21827]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:40.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:40.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:40.773 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:40.774 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:40.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[26293]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:40.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:40.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:40.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:40.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:40 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:40.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:40 smithi156 ceph-mon[22679]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:40.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:40 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:40.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:40 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:40.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:40 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:40.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:40 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:42.450 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 1 on host 'smithi070' 2024-04-24T16:17:42.745 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:42 smithi070 ceph-mon[21827]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:42.745 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:42 smithi070 ceph-mon[26293]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:42.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:42 smithi156 ceph-mon[22679]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:43.098 DEBUG:teuthology.orchestra.run.smithi070:osd.1> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.1.service 2024-04-24T16:17:43.100 INFO:tasks.cephadm:Deploying osd.2 on smithi070 with /dev/vg_nvme/lv_2... 2024-04-24T16:17:43.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[21827]: from='osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[21827]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[26293]: from='osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[26293]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:17:43.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:43 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:43.809 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:43 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:43.809 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:43 smithi156 ceph-mon[22679]: from='osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:17:43.809 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:43 smithi156 ceph-mon[22679]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:17:43.809 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:43 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:44.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:44 smithi156 ceph-mon[22679]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:44.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:44 smithi156 ceph-mon[22679]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T16:17:44.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:44 smithi156 ceph-mon[22679]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T16:17:44.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:44 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:44.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:44 smithi156 ceph-mon[22679]: from='osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:44.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:44 smithi156 ceph-mon[22679]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:44.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[21827]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[21827]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[21827]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[21827]: from='osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[21827]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[26293]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[26293]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[26293]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:44.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[26293]: from='osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:44.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-mon[26293]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:17:44.725 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[32158]: 2024-04-24T16:17:44.462+0000 7f2acaf91700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T16:17:44.725 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:17:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[32158]: 2024-04-24T16:17:44.465+0000 7f2ac3924700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:17:45.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[26293]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:17:45.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[26293]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T16:17:45.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:45.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:45.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[21827]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:17:45.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[21827]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T16:17:45.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:45.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:45 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:45 smithi156 ceph-mon[22679]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:17:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:45 smithi156 ceph-mon[22679]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T16:17:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:45 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:45 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:46.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:17:46.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:17:46.698 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144] boot 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:46.699 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144] boot 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:46.700 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:46.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:46.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:46.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:46.701 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:46 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: osd.1 [v2:172.21.15.70:6810/2849413144,v1:172.21.15.70:6811/2849413144] boot 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:46.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:17:46.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:46.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:17:46.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:46.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:17:46.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:46 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:17:47.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:47 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi070 2024-04-24T16:17:47.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:47 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi070 to 8032M 2024-04-24T16:17:47.783 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:47 smithi070 ceph-mon[26293]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T16:17:47.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:47 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi070 2024-04-24T16:17:47.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:47 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi070 to 8032M 2024-04-24T16:17:47.783 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:47 smithi070 ceph-mon[21827]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T16:17:47.788 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:17:47.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2024-04-24T16:17:47.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:47 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi070 2024-04-24T16:17:47.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:47 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi070 to 8032M 2024-04-24T16:17:47.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:47 smithi156 ceph-mon[22679]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T16:17:48.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:48 smithi070 ceph-mon[21827]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:48.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:48 smithi070 ceph-mon[26293]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:48.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:48 smithi156 ceph-mon[22679]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:50.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[26293]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:50.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[26293]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:50.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:50.711 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:50.712 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:50.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[21827]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:50.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[21827]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:50.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:50.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:50.712 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:50.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:50 smithi156 ceph-mon[22679]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:50.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:50 smithi156 ceph-mon[22679]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:17:50.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:50 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:17:50.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:50 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:17:50.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:50 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:52 smithi156 ceph-mon[22679]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:52 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2167407892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]: dispatch 2024-04-24T16:17:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:52 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]: dispatch 2024-04-24T16:17:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:52 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]': finished 2024-04-24T16:17:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:52 smithi156 ceph-mon[22679]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T16:17:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:52 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:17:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[21827]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2167407892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]: dispatch 2024-04-24T16:17:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]: dispatch 2024-04-24T16:17:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]': finished 2024-04-24T16:17:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[21827]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T16:17:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:17:53.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[26293]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:53.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2167407892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]: dispatch 2024-04-24T16:17:53.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]: dispatch 2024-04-24T16:17:53.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22be76f9-af84-4147-9dd5-99e47e3b3ea4"}]': finished 2024-04-24T16:17:53.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[26293]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T16:17:53.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:52 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:17:53.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:53 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1319318393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:54.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:53 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1319318393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:54.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:53 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1319318393' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:17:54.848 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:54 smithi070 ceph-mon[26293]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:54.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:54 smithi070 ceph-mon[21827]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:54.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:54 smithi156 ceph-mon[22679]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:56.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:56 smithi070 ceph-mon[21827]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:56.770 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:56 smithi070 ceph-mon[26293]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:56.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:56 smithi156 ceph-mon[22679]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:58.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:58 smithi070 ceph-mon[21827]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:58.683 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:17:58.683 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:58.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:58 smithi070 ceph-mon[26293]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:58.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:17:58.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:58.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:58 smithi156 ceph-mon[22679]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:17:58.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:17:58.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:17:59.675 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:17:59 smithi070 ceph-mon[26293]: Deploying daemon osd.2 on smithi070 2024-04-24T16:17:59.675 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:17:59 smithi070 ceph-mon[21827]: Deploying daemon osd.2 on smithi070 2024-04-24T16:17:59.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:17:59 smithi156 ceph-mon[22679]: Deploying daemon osd.2 on smithi070 2024-04-24T16:18:00.852 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:00 smithi070 ceph-mon[26293]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:00.852 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:00 smithi070 ceph-mon[21827]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:00.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:00 smithi156 ceph-mon[22679]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:02.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:02 smithi070 ceph-mon[21827]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:02.848 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:02 smithi070 ceph-mon[26293]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:02.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:02 smithi156 ceph-mon[22679]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:04.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[26293]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:04.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:04.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:04.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:04.256 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:04.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[21827]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:04.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:04.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:04.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:04.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:04 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:04.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:04 smithi156 ceph-mon[22679]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:04.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:04 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:04.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:04 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:04.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:04 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:04.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:04 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:05.238 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:18:05.239 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:18:05.239 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:18:05.239 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:18:05.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:18:05.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:18:05.937 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 2 on host 'smithi070' 2024-04-24T16:18:06.251 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:06 smithi070 ceph-mon[26293]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:06.251 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:06 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:06.251 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:06 smithi070 ceph-mon[21827]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:06.251 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:06 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:06.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:06 smithi156 ceph-mon[22679]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:06.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:06 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:06.553 DEBUG:teuthology.orchestra.run.smithi070:osd.2> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.2.service 2024-04-24T16:18:06.555 INFO:tasks.cephadm:Deploying osd.3 on smithi070 with /dev/vg_nvme/lv_1... 2024-04-24T16:18:06.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-04-24T16:18:07.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:07 smithi070 ceph-mon[26293]: from='osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:18:07.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:07 smithi070 ceph-mon[26293]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:18:07.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:07 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:07.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:07 smithi070 ceph-mon[21827]: from='osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:18:07.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:07 smithi070 ceph-mon[21827]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:18:07.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:07 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:07.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:07 smithi156 ceph-mon[22679]: from='osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:18:07.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:07 smithi156 ceph-mon[22679]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:18:07.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:07 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:08.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[26293]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:08.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[26293]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T16:18:08.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[26293]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T16:18:08.178 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[26293]: from='osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[26293]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[21827]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[21827]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[21827]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[21827]: from='osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:08.179 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-mon[21827]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:08.179 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[35098]: 2024-04-24T16:18:08.020+0000 7fc610924700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T16:18:08.180 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:08 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[35098]: 2024-04-24T16:18:08.023+0000 7fc60cabe700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:18:08.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:08 smithi156 ceph-mon[22679]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:08.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:08 smithi156 ceph-mon[22679]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T16:18:08.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:08 smithi156 ceph-mon[22679]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T16:18:08.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:08 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:08.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:08 smithi156 ceph-mon[22679]: from='osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:08.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:08 smithi156 ceph-mon[22679]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:09.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[21827]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:18:09.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[21827]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T16:18:09.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:09.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:09.167 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[26293]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:18:09.167 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[26293]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T16:18:09.167 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:09.168 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:09 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:09.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:09 smithi156 ceph-mon[22679]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:18:09.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:09 smithi156 ceph-mon[22679]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T16:18:09.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:09 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:09.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:09 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[21827]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[21827]: osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873] boot 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[21827]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:18:10.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:18:10.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[26293]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:10.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[26293]: osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873] boot 2024-04-24T16:18:10.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[26293]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T16:18:10.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:10 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:10.352 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:18:10.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2024-04-24T16:18:10.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:10 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:18:10.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:10 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:18:10.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:10 smithi156 ceph-mon[22679]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-24T16:18:10.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:10 smithi156 ceph-mon[22679]: osd.2 [v2:172.21.15.70:6818/679611873,v1:172.21.15.70:6819/679611873] boot 2024-04-24T16:18:10.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:10 smithi156 ceph-mon[22679]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T16:18:10.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:10 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:18:11.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:11 smithi070 ceph-mon[21827]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T16:18:11.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:11 smithi070 ceph-mon[26293]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T16:18:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:11 smithi156 ceph-mon[22679]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:12 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:12.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-24T16:18:12.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-24T16:18:12.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T16:18:12.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T16:18:12.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:12.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:12 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:13.041 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:18:12 smithi070 sudo[36414]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T16:18:13.041 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:18:12 smithi070 sudo[36414]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:13.041 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:18:12 smithi070 sudo[36414]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:13.368 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi070 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi070 to 5354M 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi070 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi070 to 5354M 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T16:18:13.369 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T16:18:13.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:13.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:13.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:13.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:13 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:13.370 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36439]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T16:18:13.370 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36439]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:13.370 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36439]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:13.370 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36479]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T16:18:13.559 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi070 2024-04-24T16:18:13.559 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi070 to 5354M 2024-04-24T16:18:13.559 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T16:18:13.559 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T16:18:13.559 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:13.559 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:13.559 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:13.560 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:13 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:13.632 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36479]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:13.633 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36479]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:13.885 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36525]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T16:18:13.885 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36525]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:13.885 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36525]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:14.140 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36548]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T16:18:14.140 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36548]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:14.140 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:13 smithi070 sudo[36548]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:14.398 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36553]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T16:18:14.398 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36553]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:14.398 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36553]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:14.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36556]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T16:18:14.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36556]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:14.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36556]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:14.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36578]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T16:18:14.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36578]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:14.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36578]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:14.901 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[26293]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:14.901 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[26293]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:14.901 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[26293]: osdmap e22: 3 total, 3 up, 3 in 2024-04-24T16:18:14.901 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[26293]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-24T16:18:14.901 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[26293]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:14.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[21827]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:14.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[21827]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:14.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[21827]: osdmap e22: 3 total, 3 up, 3 in 2024-04-24T16:18:14.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[21827]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-24T16:18:14.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:14 smithi070 ceph-mon[21827]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:14.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:14 smithi156 ceph-mon[22679]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:14 smithi156 ceph-mon[22679]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:14 smithi156 ceph-mon[22679]: osdmap e22: 3 total, 3 up, 3 in 2024-04-24T16:18:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:14 smithi156 ceph-mon[22679]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-24T16:18:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:14 smithi156 ceph-mon[22679]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:15.163 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36588]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T16:18:15.163 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36588]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:15.163 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:14 smithi070 sudo[36588]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:15.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 sudo[36600]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T16:18:15.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 sudo[36600]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:15.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 sudo[36600]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:15.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 sudo[24193]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T16:18:15.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 sudo[24193]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:15.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 sudo[24193]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:15.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 sudo[24196]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T16:18:15.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 sudo[24196]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-24T16:18:15.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 sudo[24196]: pam_unix(sudo:session): session closed for user root 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3427787177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2f8f93a-5321-4567-9551-cd2b1aa28691"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:15 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3427787177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2f8f93a-5321-4567-9551-cd2b1aa28691"}]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:18:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:18:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:18:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:15 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3427787177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2f8f93a-5321-4567-9551-cd2b1aa28691"}]: dispatch 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3427787177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2f8f93a-5321-4567-9551-cd2b1aa28691"}]': finished 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2911127429' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3427787177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2f8f93a-5321-4567-9551-cd2b1aa28691"}]': finished 2024-04-24T16:18:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T16:18:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:16 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2911127429' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3427787177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2f8f93a-5321-4567-9551-cd2b1aa28691"}]': finished 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:16 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2911127429' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:18:18.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:18 smithi070 ceph-mon[21827]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:18.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:18 smithi070 ceph-mon[26293]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:19.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:18 smithi156 ceph-mon[22679]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:20.757 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:20 smithi070 ceph-mon[26293]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:20.757 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:20 smithi070 ceph-mon[21827]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:21.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:20 smithi156 ceph-mon[22679]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:22.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[26293]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:22.916 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:18:22.916 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:22.916 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[26293]: Deploying daemon osd.3 on smithi070 2024-04-24T16:18:22.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[21827]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:22.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:18:22.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:22.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:22 smithi070 ceph-mon[21827]: Deploying daemon osd.3 on smithi070 2024-04-24T16:18:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:22 smithi156 ceph-mon[22679]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:22 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:18:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:22 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:22 smithi156 ceph-mon[22679]: Deploying daemon osd.3 on smithi070 2024-04-24T16:18:24.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:24 smithi070 ceph-mon[21827]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:24.964 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:24 smithi070 ceph-mon[26293]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:25.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:24 smithi156 ceph-mon[22679]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:26.875 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:26 smithi070 ceph-mon[26293]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:26.875 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:26 smithi070 ceph-mon[21827]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:26.929 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:26 smithi156 ceph-mon[22679]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:27 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:27 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:27 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:27 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:28.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:28.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:28.189 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:28.190 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:28.190 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:28.190 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:28.190 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:28.190 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:27 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:29.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:28 smithi156 ceph-mon[22679]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:29.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:28 smithi070 ceph-mon[26293]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:29.280 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:28 smithi070 ceph-mon[21827]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:29.721 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 3 on host 'smithi070' 2024-04-24T16:18:30.331 DEBUG:teuthology.orchestra.run.smithi070:osd.3> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.3.service 2024-04-24T16:18:30.333 INFO:tasks.cephadm:Deploying osd.4 on smithi156 with /dev/vg_nvme/lv_4... 2024-04-24T16:18:30.333 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-04-24T16:18:30.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:30 smithi156 ceph-mon[22679]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:30.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:30 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:30.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:30 smithi156 ceph-mon[22679]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:18:31.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:30 smithi070 ceph-mon[26293]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:31.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:30 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:31.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:30 smithi070 ceph-mon[26293]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:18:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:30 smithi070 ceph-mon[21827]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:30 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:31.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:30 smithi070 ceph-mon[21827]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:18:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:31 smithi156 ceph-mon[22679]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T16:18:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:31 smithi156 ceph-mon[22679]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T16:18:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:31 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:31 smithi156 ceph-mon[22679]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:31 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[21827]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T16:18:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[21827]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T16:18:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[21827]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:32.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[26293]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T16:18:32.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[26293]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T16:18:32.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:32.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[26293]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:18:32.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:32.292 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[38072]: 2024-04-24T16:18:31.845+0000 7f4785d2b700 -1 osd.3 0 waiting for initial osdmap 2024-04-24T16:18:32.292 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:18:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[38072]: 2024-04-24T16:18:31.849+0000 7f477eebf700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:18:32.804 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:18:32.823 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2024-04-24T16:18:33.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[21827]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[21827]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[21827]: osdmap e25: 4 total, 3 up, 4 in 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[26293]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[26293]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[26293]: osdmap e25: 4 total, 3 up, 4 in 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:32 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:33.131 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:32 smithi156 ceph-mon[22679]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:33.131 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:32 smithi156 ceph-mon[22679]: from='osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-24T16:18:33.131 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:32 smithi156 ceph-mon[22679]: osdmap e25: 4 total, 3 up, 4 in 2024-04-24T16:18:33.131 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:32 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:33.131 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:32 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:34.115 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:18:34.115 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:18:34.115 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[26293]: osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519] boot 2024-04-24T16:18:34.115 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[26293]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T16:18:34.115 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:34.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:18:34.116 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:18:34.116 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[21827]: osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519] boot 2024-04-24T16:18:34.116 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[21827]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T16:18:34.116 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:33 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:34.139 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:33 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:18:34.139 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:33 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:18:34.139 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:33 smithi156 ceph-mon[22679]: osd.3 [v2:172.21.15.70:6826/1561941519,v1:172.21.15.70:6827/1561941519] boot 2024-04-24T16:18:34.139 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:33 smithi156 ceph-mon[22679]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T16:18:34.139 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:33 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:18:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T16:18:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:35.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:34 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:34 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:36.084 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:35 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi070 2024-04-24T16:18:36.084 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:35 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi070 to 4016M 2024-04-24T16:18:36.084 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:35 smithi156 ceph-mon[22679]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:36.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:35 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi070 2024-04-24T16:18:36.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:35 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi070 to 4016M 2024-04-24T16:18:36.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:35 smithi070 ceph-mon[21827]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:36.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:35 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi070 2024-04-24T16:18:36.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:35 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi070 to 4016M 2024-04-24T16:18:36.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:35 smithi070 ceph-mon[26293]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:36 smithi156 ceph-mon[22679]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:36 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/405648746' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]: dispatch 2024-04-24T16:18:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:36 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]: dispatch 2024-04-24T16:18:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:36 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]': finished 2024-04-24T16:18:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:36 smithi156 ceph-mon[22679]: osdmap e28: 5 total, 4 up, 5 in 2024-04-24T16:18:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:36 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:37.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[26293]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/405648746' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]: dispatch 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]: dispatch 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]': finished 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[26293]: osdmap e28: 5 total, 4 up, 5 in 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[21827]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/405648746' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]: dispatch 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]: dispatch 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75d4ec4e-da7f-43d7-87e4-72ab045af699"}]': finished 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[21827]: osdmap e28: 5 total, 4 up, 5 in 2024-04-24T16:18:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:36 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:37 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/1484909559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:18:38.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:37 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/1484909559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:18:38.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:37 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/1484909559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:18:39.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:38 smithi156 ceph-mon[22679]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:39.192 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:38 smithi070 ceph-mon[26293]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:39.192 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:38 smithi070 ceph-mon[21827]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:40.919 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:40 smithi156 ceph-mon[22679]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:41.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:40 smithi070 ceph-mon[21827]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:41.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:40 smithi070 ceph-mon[26293]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:43.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:42 smithi156 ceph-mon[22679]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:43.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:18:43.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:43.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:42 smithi070 ceph-mon[26293]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:43.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:18:43.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:43.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:42 smithi070 ceph-mon[21827]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:43.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:18:43.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:44.129 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:43 smithi156 ceph-mon[22679]: Deploying daemon osd.4 on smithi156 2024-04-24T16:18:44.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:43 smithi070 ceph-mon[26293]: Deploying daemon osd.4 on smithi156 2024-04-24T16:18:44.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:43 smithi070 ceph-mon[21827]: Deploying daemon osd.4 on smithi156 2024-04-24T16:18:45.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:44 smithi156 ceph-mon[22679]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:45.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:44 smithi070 ceph-mon[26293]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:45.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:44 smithi070 ceph-mon[21827]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:47.121 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:46 smithi156 ceph-mon[22679]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:47.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:46 smithi070 ceph-mon[26293]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:47.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:46 smithi070 ceph-mon[21827]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:48.642 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:48 smithi156 ceph-mon[22679]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:48.642 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:48 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:48.643 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:48 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:48.643 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:48 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:48.643 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:48 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[26293]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[21827]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:48.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:48.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:48 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:50.323 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 4 on host 'smithi156' 2024-04-24T16:18:50.613 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:50 smithi156 ceph-mon[22679]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:50.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:50 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:50.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:50 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:50.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:50 smithi070 ceph-mon[21827]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:50.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:50.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:50.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:50 smithi070 ceph-mon[26293]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:50.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:50.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:50.896 DEBUG:teuthology.orchestra.run.smithi156:osd.4> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.4.service 2024-04-24T16:18:50.898 INFO:tasks.cephadm:Deploying osd.5 on smithi156 with /dev/vg_nvme/lv_3... 2024-04-24T16:18:50.899 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-04-24T16:18:51.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:51 smithi156 ceph-mon[22679]: from='osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:18:51.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:51 smithi156 ceph-mon[22679]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:18:51.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:51 smithi070 ceph-mon[21827]: from='osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:18:51.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:51 smithi070 ceph-mon[21827]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:18:51.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:51 smithi070 ceph-mon[26293]: from='osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:18:51.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:51 smithi070 ceph-mon[26293]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:18:52.644 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-mon[22679]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:52.644 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-mon[22679]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T16:18:52.644 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-mon[22679]: osdmap e29: 5 total, 4 up, 5 in 2024-04-24T16:18:52.644 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:52.644 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-mon[22679]: from='osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:18:52.644 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-mon[22679]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:18:52.644 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[26116]: 2024-04-24T16:18:52.384+0000 7f8acd68a700 -1 osd.4 0 waiting for initial osdmap 2024-04-24T16:18:52.644 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:18:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[26116]: 2024-04-24T16:18:52.388+0000 7f8ac601d700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:18:52.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[21827]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:52.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[21827]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T16:18:52.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[21827]: osdmap e29: 5 total, 4 up, 5 in 2024-04-24T16:18:52.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:52.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[21827]: from='osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:18:52.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[21827]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:18:52.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[26293]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:52.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[26293]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T16:18:52.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[26293]: osdmap e29: 5 total, 4 up, 5 in 2024-04-24T16:18:52.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:52.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[26293]: from='osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:18:52.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:52 smithi070 ceph-mon[26293]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:18:53.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:53 smithi156 ceph-mon[22679]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:18:53.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:53 smithi156 ceph-mon[22679]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T16:18:53.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:53 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:53 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:53 smithi156 ceph-mon[22679]: osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868] boot 2024-04-24T16:18:53.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:53 smithi156 ceph-mon[22679]: osdmap e31: 5 total, 5 up, 5 in 2024-04-24T16:18:53.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:53 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[21827]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[21827]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[21827]: osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868] boot 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[21827]: osdmap e31: 5 total, 5 up, 5 in 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[26293]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[26293]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T16:18:53.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:53.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[26293]: osd.4 [v2:172.21.15.156:6800/1959136868,v1:172.21.15.156:6801/1959136868] boot 2024-04-24T16:18:53.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[26293]: osdmap e31: 5 total, 5 up, 5 in 2024-04-24T16:18:53.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:53 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:18:54.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:54 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:18:54.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:54 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:18:54.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:54 smithi156 ceph-mon[22679]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:54.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:54 smithi156 ceph-mon[22679]: osdmap e32: 5 total, 5 up, 5 in 2024-04-24T16:18:54.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:18:54.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:18:54.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[21827]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:54.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[21827]: osdmap e32: 5 total, 5 up, 5 in 2024-04-24T16:18:54.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:18:54.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:18:54.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[26293]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-24T16:18:54.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:54 smithi070 ceph-mon[26293]: osdmap e32: 5 total, 5 up, 5 in 2024-04-24T16:18:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi156 2024-04-24T16:18:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-24T16:18:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:55 smithi156 ceph-mon[22679]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T16:18:55.931 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:18:55.949 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi156 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[21827]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi156 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi156 to 17088M 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:18:56.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:18:56.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:55 smithi070 ceph-mon[26293]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T16:18:56.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:56 smithi156 ceph-mon[22679]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:18:56.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:56 smithi156 ceph-mon[22679]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T16:18:57.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:56 smithi070 ceph-mon[21827]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:18:57.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:56 smithi070 ceph-mon[21827]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T16:18:57.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:56 smithi070 ceph-mon[26293]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:18:57.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:56 smithi070 ceph-mon[26293]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T16:18:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:57 smithi156 ceph-mon[22679]: osdmap e35: 5 total, 5 up, 5 in 2024-04-24T16:18:58.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:57 smithi070 ceph-mon[21827]: osdmap e35: 5 total, 5 up, 5 in 2024-04-24T16:18:58.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:57 smithi070 ceph-mon[26293]: osdmap e35: 5 total, 5 up, 5 in 2024-04-24T16:18:58.791 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:58 smithi156 ceph-mon[22679]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T16:18:58.791 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:58 smithi156 ceph-mon[22679]: from='client.34217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:58.791 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:58.791 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:58.791 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:18:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:59.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[21827]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[21827]: from='client.34217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[26293]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[26293]: from='client.34217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:18:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:18:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:00.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:00 smithi156 ceph-mon[22679]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T16:19:00.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:00 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/1933560325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]: dispatch 2024-04-24T16:19:00.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:00 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]: dispatch 2024-04-24T16:19:00.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:00 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]': finished 2024-04-24T16:19:00.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:00 smithi156 ceph-mon[22679]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T16:19:00.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:01.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[21827]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/1933560325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]: dispatch 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]: dispatch 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]': finished 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[21827]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[26293]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/1933560325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]: dispatch 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]: dispatch 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ccc7fa7c-53c3-46ae-af1f-44c1bee32330"}]': finished 2024-04-24T16:19:01.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[26293]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T16:19:01.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:01.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:01 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/885436034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:02.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:01 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/885436034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:02.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:01 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/885436034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:02.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:02 smithi156 ceph-mon[22679]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T16:19:03.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:02 smithi070 ceph-mon[21827]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T16:19:03.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:02 smithi070 ceph-mon[26293]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-24T16:19:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:04 smithi156 ceph-mon[22679]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-24T16:19:05.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:04 smithi070 ceph-mon[21827]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-24T16:19:05.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:04 smithi070 ceph-mon[26293]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-24T16:19:05.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:19:05.628 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:19:06.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:19:06.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:19:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:19:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:19:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:06 smithi156 ceph-mon[22679]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:06 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:19:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:06 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:06 smithi156 ceph-mon[22679]: Deploying daemon osd.5 on smithi156 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[21827]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[21827]: Deploying daemon osd.5 on smithi156 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[26293]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:06 smithi070 ceph-mon[26293]: Deploying daemon osd.5 on smithi156 2024-04-24T16:19:08.901 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:08 smithi156 ceph-mon[22679]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:09.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:08 smithi070 ceph-mon[21827]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:09.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:08 smithi070 ceph-mon[26293]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:10.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:10 smithi156 ceph-mon[22679]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:11.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:10 smithi070 ceph-mon[26293]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:11.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:10 smithi070 ceph-mon[21827]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:12.004 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:11 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:12.004 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:11 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:12.004 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:11 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:12.005 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:11 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:12.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:12.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:12.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:12.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:12.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:12.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:12.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:12.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:11 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:13.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:12 smithi070 ceph-mon[21827]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:13.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:12 smithi070 ceph-mon[26293]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:13.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:12 smithi156 ceph-mon[22679]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:13.425 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 5 on host 'smithi156' 2024-04-24T16:19:14.037 DEBUG:teuthology.orchestra.run.smithi156:osd.5> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.5.service 2024-04-24T16:19:14.039 INFO:tasks.cephadm:Deploying osd.6 on smithi156 with /dev/vg_nvme/lv_2... 2024-04-24T16:19:14.039 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-04-24T16:19:14.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:14 smithi156 ceph-mon[22679]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:14.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:14 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:14.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:14 smithi156 ceph-mon[22679]: from='osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:19:14.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:14 smithi156 ceph-mon[22679]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:19:14.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:14 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:14.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[21827]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:14.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:14.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[21827]: from='osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:19:14.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[21827]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:19:14.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[26293]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:14.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[26293]: from='osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:19:14.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[26293]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:19:14.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:14 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:16.003 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:15 smithi156 ceph-mon[22679]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T16:19:16.004 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:15 smithi156 ceph-mon[22679]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T16:19:16.004 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:15 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:16.004 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:15 smithi156 ceph-mon[22679]: from='osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:16.004 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:15 smithi156 ceph-mon[22679]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:16.004 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:19:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[29004]: 2024-04-24T16:19:15.699+0000 7fa307b6e700 -1 osd.5 0 waiting for initial osdmap 2024-04-24T16:19:16.004 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:19:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[29004]: 2024-04-24T16:19:15.704+0000 7fa303507700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[21827]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[21827]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[21827]: from='osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[21827]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[26293]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[26293]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T16:19:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[26293]: from='osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:15 smithi070 ceph-mon[26293]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[21827]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[21827]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[21827]: osdmap e38: 6 total, 5 up, 6 in 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[26293]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[26293]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[26293]: osdmap e38: 6 total, 5 up, 6 in 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:16 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:17.127 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:16 smithi156 ceph-mon[22679]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-24T16:19:17.127 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:16 smithi156 ceph-mon[22679]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:19:17.127 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:16 smithi156 ceph-mon[22679]: osdmap e38: 6 total, 5 up, 6 in 2024-04-24T16:19:17.127 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:16 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:17.127 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:16 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:17.133 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:19:17.155 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2024-04-24T16:19:17.940 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:17 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:19:17.941 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:17 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:19:17.941 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:17 smithi156 ceph-mon[22679]: osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227] boot 2024-04-24T16:19:17.941 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:17 smithi156 ceph-mon[22679]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T16:19:17.941 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:17 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:17.941 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:17 smithi156 ceph-mon[22679]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[21827]: osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227] boot 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[21827]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[21827]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[26293]: osd.5 [v2:172.21.15.156:6808/4243847227,v1:172.21.15.156:6809/4243847227] boot 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[26293]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T16:19:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:19:18.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:17 smithi070 ceph-mon[26293]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T16:19:18.817 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:18 smithi156 ceph-mon[22679]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:18.817 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:18 smithi156 ceph-mon[22679]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T16:19:19.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:18 smithi070 ceph-mon[21827]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:19.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:18 smithi070 ceph-mon[21827]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T16:19:19.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:18 smithi070 ceph-mon[26293]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:19.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:18 smithi070 ceph-mon[26293]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T16:19:20.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi156 2024-04-24T16:19:20.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:20.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:20.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-24T16:19:20.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:20.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:20.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:20.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T16:19:20.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:19:20.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:19:20.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:19 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi156 2024-04-24T16:19:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-24T16:19:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:20.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi156 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi156 to 8544M 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:19:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:19:20.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:19 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:20.956 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:20 smithi156 ceph-mon[22679]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:20.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:20 smithi156 ceph-mon[22679]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:19:20.957 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:20 smithi156 ceph-mon[22679]: osdmap e43: 6 total, 6 up, 6 in 2024-04-24T16:19:21.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:20 smithi070 ceph-mon[21827]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:21.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:20 smithi070 ceph-mon[21827]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:19:21.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:20 smithi070 ceph-mon[21827]: osdmap e43: 6 total, 6 up, 6 in 2024-04-24T16:19:21.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:20 smithi070 ceph-mon[26293]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:21.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:20 smithi070 ceph-mon[26293]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:19:21.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:20 smithi070 ceph-mon[26293]: osdmap e43: 6 total, 6 up, 6 in 2024-04-24T16:19:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:22 smithi156 ceph-mon[22679]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:22 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/424530032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]: dispatch 2024-04-24T16:19:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:22 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]: dispatch 2024-04-24T16:19:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:22 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]': finished 2024-04-24T16:19:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:22 smithi156 ceph-mon[22679]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T16:19:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:22 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:22 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/2187581286' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:23.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[21827]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/424530032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]: dispatch 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]: dispatch 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]': finished 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[21827]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/2187581286' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[26293]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/424530032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]: dispatch 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]: dispatch 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d423492-ffef-466d-afa8-ea869cf99ced"}]': finished 2024-04-24T16:19:23.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[26293]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T16:19:23.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:23.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:22 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/2187581286' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:24.990 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:24 smithi156 ceph-mon[22679]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:25.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:24 smithi070 ceph-mon[21827]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:25.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:24 smithi070 ceph-mon[26293]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:26 smithi156 ceph-mon[22679]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:27.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:26 smithi070 ceph-mon[21827]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:27.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:26 smithi070 ceph-mon[26293]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:28 smithi156 ceph-mon[22679]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:28 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:19:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:28 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:28 smithi156 ceph-mon[22679]: Deploying daemon osd.6 on smithi156 2024-04-24T16:19:29.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[21827]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:29.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:19:29.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:29.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[21827]: Deploying daemon osd.6 on smithi156 2024-04-24T16:19:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[26293]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:19:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:28 smithi070 ceph-mon[26293]: Deploying daemon osd.6 on smithi156 2024-04-24T16:19:31.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:30 smithi156 ceph-mon[22679]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:31.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:30 smithi070 ceph-mon[21827]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:31.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:30 smithi070 ceph-mon[26293]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:33.067 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[21827]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:33.067 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[26293]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:33.068 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:32 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:33.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:32 smithi156 ceph-mon[22679]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:33.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:32 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:33.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:32 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:33.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:32 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:33.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:32 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:34 smithi156 ceph-mon[22679]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:35.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:34 smithi070 ceph-mon[26293]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:34 smithi070 ceph-mon[21827]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:35.516 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 6 on host 'smithi156' 2024-04-24T16:19:36.475 DEBUG:teuthology.orchestra.run.smithi156:osd.6> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.6.service 2024-04-24T16:19:36.477 INFO:tasks.cephadm:Deploying osd.7 on smithi156 with /dev/vg_nvme/lv_1... 2024-04-24T16:19:36.477 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[26293]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[26293]: from='osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[26293]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[21827]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[21827]: from='osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:19:36.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:36 smithi070 ceph-mon[21827]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:19:36.846 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:36 smithi156 ceph-mon[22679]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:36.847 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:36 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:36.847 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:36 smithi156 ceph-mon[22679]: from='osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:19:36.847 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:36 smithi156 ceph-mon[22679]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:19:37.626 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[31875]: 2024-04-24T16:19:37.529+0000 7f91d8538700 -1 osd.6 0 waiting for initial osdmap 2024-04-24T16:19:37.627 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[31875]: 2024-04-24T16:19:37.535+0000 7f91d3ed1700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:19:37.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-mon[22679]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T16:19:37.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-mon[22679]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T16:19:37.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:37.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-mon[22679]: from='osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:37.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-mon[22679]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:37.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:37 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:37.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[26293]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T16:19:37.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[26293]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T16:19:37.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:37.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[26293]: from='osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:37.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[26293]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:37.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:37.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[21827]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T16:19:37.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[21827]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T16:19:37.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:37.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[21827]: from='osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:37.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[21827]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:19:37.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:37 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: osdmap e46: 7 total, 6 up, 7 in 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544] boot 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T16:19:38.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:38 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: osdmap e46: 7 total, 6 up, 7 in 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544] boot 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T16:19:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: osdmap e46: 7 total, 6 up, 7 in 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: osd.6 [v2:172.21.15.156:6816/1319206544,v1:172.21.15.156:6817/1319206544] boot 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T16:19:38.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:38 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:19:39.693 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:39 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:19:39.693 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:39 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:19:39.693 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:39 smithi156 ceph-mon[22679]: osdmap e48: 7 total, 7 up, 7 in 2024-04-24T16:19:40.009 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T16:19:40.028 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2024-04-24T16:19:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:39 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:19:40.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:39 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:19:40.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:39 smithi070 ceph-mon[26293]: osdmap e48: 7 total, 7 up, 7 in 2024-04-24T16:19:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:39 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:19:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:39 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:19:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:39 smithi070 ceph-mon[21827]: osdmap e48: 7 total, 7 up, 7 in 2024-04-24T16:19:40.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:40 smithi156 ceph-mon[22679]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:40.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:40 smithi156 ceph-mon[22679]: osdmap e49: 7 total, 7 up, 7 in 2024-04-24T16:19:41.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:40 smithi070 ceph-mon[21827]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:41.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:40 smithi070 ceph-mon[21827]: osdmap e49: 7 total, 7 up, 7 in 2024-04-24T16:19:41.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:40 smithi070 ceph-mon[26293]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:41.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:40 smithi070 ceph-mon[26293]: osdmap e49: 7 total, 7 up, 7 in 2024-04-24T16:19:41.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:41 smithi156 ceph-mon[22679]: osdmap e50: 7 total, 7 up, 7 in 2024-04-24T16:19:42.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:41 smithi070 ceph-mon[21827]: osdmap e50: 7 total, 7 up, 7 in 2024-04-24T16:19:42.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:41 smithi070 ceph-mon[26293]: osdmap e50: 7 total, 7 up, 7 in 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: osdmap e51: 7 total, 7 up, 7 in 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi156 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:42 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:43.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: osdmap e51: 7 total, 7 up, 7 in 2024-04-24T16:19:43.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi156 2024-04-24T16:19:43.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:43.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: osdmap e51: 7 total, 7 up, 7 in 2024-04-24T16:19:43.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi156 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi156 to 5696M 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:42 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:43.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:43 smithi156 ceph-mon[22679]: from='client.34271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:19:43.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:43 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:19:43.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:43 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:19:43.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:43 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:44.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[26293]: from='client.34271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:19:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:19:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:19:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[21827]: from='client.34271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:19:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T16:19:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T16:19:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:43 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:44.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:44 smithi156 ceph-mon[22679]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:45.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:44 smithi070 ceph-mon[26293]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:45.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:44 smithi070 ceph-mon[21827]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:45 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/3999716349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]: dispatch 2024-04-24T16:19:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:45 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]: dispatch 2024-04-24T16:19:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:45 smithi156 ceph-mon[22679]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]': finished 2024-04-24T16:19:45.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:45 smithi156 ceph-mon[22679]: osdmap e52: 8 total, 7 up, 8 in 2024-04-24T16:19:45.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:45 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:19:45.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:45 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/1550289597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/3999716349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]: dispatch 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]: dispatch 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[21827]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]': finished 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[21827]: osdmap e52: 8 total, 7 up, 8 in 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/1550289597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/3999716349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]: dispatch 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]: dispatch 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[26293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bad0ad0-a88c-4f1c-8010-35b7fbabe791"}]': finished 2024-04-24T16:19:46.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[26293]: osdmap e52: 8 total, 7 up, 8 in 2024-04-24T16:19:46.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:19:46.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:45 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/1550289597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T16:19:47.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:46 smithi156 ceph-mon[22679]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T16:19:47.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:46 smithi070 ceph-mon[21827]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T16:19:47.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:46 smithi070 ceph-mon[26293]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T16:19:49.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:48 smithi156 ceph-mon[22679]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-24T16:19:49.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:48 smithi070 ceph-mon[21827]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-24T16:19:49.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:48 smithi070 ceph-mon[26293]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-24T16:19:51.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:50 smithi156 ceph-mon[22679]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T16:19:51.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:50 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:19:51.060 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:50 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:51.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:50 smithi070 ceph-mon[21827]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T16:19:51.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:19:51.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:50 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:51.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:50 smithi070 ceph-mon[26293]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T16:19:51.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:19:51.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:50 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:52.103 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:51 smithi156 ceph-mon[22679]: Deploying daemon osd.7 on smithi156 2024-04-24T16:19:52.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:51 smithi070 ceph-mon[21827]: Deploying daemon osd.7 on smithi156 2024-04-24T16:19:52.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:51 smithi070 ceph-mon[26293]: Deploying daemon osd.7 on smithi156 2024-04-24T16:19:53.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:52 smithi156 ceph-mon[22679]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-24T16:19:53.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:52 smithi070 ceph-mon[21827]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-24T16:19:53.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:52 smithi070 ceph-mon[26293]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-24T16:19:55.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:54 smithi156 ceph-mon[22679]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-24T16:19:55.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:54 smithi070 ceph-mon[21827]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-24T16:19:55.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:54 smithi070 ceph-mon[26293]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-24T16:19:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:56 smithi156 ceph-mon[22679]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:56 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:56 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:56 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:56 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:56.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[26293]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[21827]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:56 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:58.540 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 7 on host 'smithi156' 2024-04-24T16:19:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:19:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:19:59.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:58 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:59.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:19:59.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.063 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:19:59.063 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:19:59.063 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:19:59.063 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:19:59.063 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:19:59.063 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:58 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:19:59.084 DEBUG:teuthology.orchestra.run.smithi156:osd.7> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.7.service 2024-04-24T16:19:59.087 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-24T16:19:59.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-24T16:20:00.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[26293]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T16:20:00.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[26293]: osdmap e53: 8 total, 7 up, 8 in 2024-04-24T16:20:00.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:00.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[26293]: from='osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:20:00.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[26293]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:20:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[21827]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T16:20:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[21827]: osdmap e53: 8 total, 7 up, 8 in 2024-04-24T16:20:00.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:00.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[21827]: from='osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:20:00.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:19:59 smithi070 ceph-mon[21827]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:20:00.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:59 smithi156 ceph-mon[22679]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T16:20:00.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:59 smithi156 ceph-mon[22679]: osdmap e53: 8 total, 7 up, 8 in 2024-04-24T16:20:00.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:59 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:00.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:59 smithi156 ceph-mon[22679]: from='osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:20:00.047 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:19:59 smithi156 ceph-mon[22679]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:20:00.048 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:19:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[34873]: 2024-04-24T16:19:59.787+0000 7fe1eb327700 -1 osd.7 0 waiting for initial osdmap 2024-04-24T16:20:00.048 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:19:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[34873]: 2024-04-24T16:19:59.795+0000 7fe1e74c1700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:20:01.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[21827]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[21827]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[21827]: osdmap e54: 8 total, 7 up, 8 in 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[21827]: overall HEALTH_OK 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[26293]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[26293]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[26293]: osdmap e54: 8 total, 7 up, 8 in 2024-04-24T16:20:01.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:01.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:01.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:00 smithi070 ceph-mon[26293]: overall HEALTH_OK 2024-04-24T16:20:01.131 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:20:01.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:00 smithi156 ceph-mon[22679]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:20:01.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:00 smithi156 ceph-mon[22679]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-04-24T16:20:01.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:00 smithi156 ceph-mon[22679]: osdmap e54: 8 total, 7 up, 8 in 2024-04-24T16:20:01.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:01.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:00 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:01.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:00 smithi156 ceph-mon[22679]: overall HEALTH_OK 2024-04-24T16:20:01.692 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1713975600,"num_in_osds":8,"osd_in_since":1713975584,"num_remapped_pgs":1} 2024-04-24T16:20:01.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T16:20:01.904 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:01.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:01 smithi156 ceph-mon[22679]: purged_snaps scrub starts 2024-04-24T16:20:01.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:01 smithi156 ceph-mon[22679]: purged_snaps scrub ok 2024-04-24T16:20:01.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:01 smithi156 ceph-mon[22679]: osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815] boot 2024-04-24T16:20:01.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:01 smithi156 ceph-mon[22679]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T16:20:01.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:01 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:01.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:01 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2343449645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[26293]: purged_snaps scrub starts 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[26293]: purged_snaps scrub ok 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[26293]: osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815] boot 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[26293]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2343449645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[21827]: purged_snaps scrub starts 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[21827]: purged_snaps scrub ok 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[21827]: osd.7 [v2:172.21.15.156:6824/1364928815,v1:172.21.15.156:6825/1364928815] boot 2024-04-24T16:20:02.028 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[21827]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T16:20:02.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:02.029 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:01 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2343449645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T16:20:02.959 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:20:02.959 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: osdmap e56: 8 total, 8 up, 8 in 2024-04-24T16:20:02.959 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi156 2024-04-24T16:20:02.959 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: osdmap e56: 8 total, 8 up, 8 in 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi156 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.961 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.961 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:02.961 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-24T16:20:02.961 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:02.961 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:02 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-24T16:20:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: osdmap e56: 8 total, 8 up, 8 in 2024-04-24T16:20:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: Detected new or changed devices on smithi156 2024-04-24T16:20:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:03.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:20:03.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi156 to 4272M 2024-04-24T16:20:03.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:03.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:02 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:03.609 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:20:03.609 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":57,"fsid":"be31be9a-0255-11ef-bc93-c7b262605968","created":"2024-04-24T16:15:17.566184+0000","modified":"2024-04-24T16:20:02.794734+0000","last_up_change":"2024-04-24T16:20:00.786057+0000","last_in_change":"2024-04-24T16:19:44.833161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T16:18:11.216254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1479dc88-7964-4663-a0e4-7c385281030b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6803","nonce":351334718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6805","nonce":351334718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6809","nonce":351334718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6807","nonce":351334718}]},"public_addr":"172.21.15.70:6803/351334718","cluster_addr":"172.21.15.70:6805/351334718","heartbeat_back_addr":"172.21.15.70:6809/351334718","heartbeat_front_addr":"172.21.15.70:6807/351334718","state":["exists","up"]},{"osd":1,"uuid":"48295f9b-7ffc-4b68-99b6-7599af8a0aff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6811","nonce":2849413144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6813","nonce":2849413144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6817","nonce":2849413144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6815","nonce":2849413144}]},"public_addr":"172.21.15.70:6811/2849413144","cluster_addr":"172.21.15.70:6813/2849413144","heartbeat_back_addr":"172.21.15.70:6817/2849413144","heartbeat_front_addr":"172.21.15.70:6815/2849413144","state":["exists","up"]},{"osd":2,"uuid":"22be76f9-af84-4147-9dd5-99e47e3b3ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6819","nonce":679611873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6821","nonce":679611873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6825","nonce":679611873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6823","nonce":679611873}]},"public_addr":"172.21.15.70:6819/679611873","cluster_addr":"172.21.15.70:6821/679611873","heartbeat_back_addr":"172.21.15.70:6825/679611873","heartbeat_front_addr":"172.21.15.70:6823/679611873","state":["exists","up"]},{"osd":3,"uuid":"f2f8f93a-5321-4567-9551-cd2b1aa28691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6827","nonce":1561941519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6829","nonce":1561941519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6833","nonce":1561941519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6831","nonce":1561941519}]},"public_addr":"172.21.15.70:6827/1561941519","cluster_addr":"172.21.15.70:6829/1561941519","heartbeat_back_addr":"172.21.15.70:6833/1561941519","heartbeat_front_addr":"172.21.15.70:6831/1561941519","state":["exists","up"]},{"osd":4,"uuid":"75d4ec4e-da7f-43d7-87e4-72ab045af699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6801","nonce":1959136868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6803","nonce":1959136868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6807","nonce":1959136868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6805","nonce":1959136868}]},"public_addr":"172.21.15.156:6801/1959136868","cluster_addr":"172.21.15.156:6803/1959136868","heartbeat_back_addr":"172.21.15.156:6807/1959136868","heartbeat_front_addr":"172.21.15.156:6805/1959136868","state":["exists","up"]},{"osd":5,"uuid":"ccc7fa7c-53c3-46ae-af1f-44c1bee32330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6809","nonce":4243847227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6811","nonce":4243847227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6815","nonce":4243847227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6813","nonce":4243847227}]},"public_addr":"172.21.15.156:6809/4243847227","cluster_addr":"172.21.15.156:6811/4243847227","heartbeat_back_addr":"172.21.15.156:6815/4243847227","heartbeat_front_addr":"172.21.15.156:6813/4243847227","state":["exists","up"]},{"osd":6,"uuid":"5d423492-ffef-466d-afa8-ea869cf99ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6817","nonce":1319206544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6819","nonce":1319206544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6823","nonce":1319206544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6821","nonce":1319206544}]},"public_addr":"172.21.15.156:6817/1319206544","cluster_addr":"172.21.15.156:6819/1319206544","heartbeat_back_addr":"172.21.15.156:6823/1319206544","heartbeat_front_addr":"172.21.15.156:6821/1319206544","state":["exists","up"]},{"osd":7,"uuid":"0bad0ad0-a88c-4f1c-8010-35b7fbabe791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6825","nonce":1364928815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6827","nonce":1364928815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6831","nonce":1364928815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6829","nonce":1364928815}]},"public_addr":"172.21.15.156:6825/1364928815","cluster_addr":"172.21.15.156:6827/1364928815","heartbeat_back_addr":"172.21.15.156:6831/1364928815","heartbeat_front_addr":"172.21.15.156:6829/1364928815","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:17:20.883636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:17:43.512527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:07.126511+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:30.889304+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:51.518774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:14.757678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:36.844389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:59.520858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.70:0/4043624147":"2024-04-25T16:16:04.858290+0000","172.21.15.70:6800/2548077742":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/4129610450":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/1502360754":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/1351576920":"2024-04-25T16:15:43.572541+0000","172.21.15.70:0/2897538162":"2024-04-25T16:15:43.572541+0000","172.21.15.70:6800/3814823340":"2024-04-25T16:15:43.572541+0000","172.21.15.70:6801/2548077742":"2024-04-25T16:16:04.858290+0000","172.21.15.70:6801/3814823340":"2024-04-25T16:15:43.572541+0000","172.21.15.70:0/2369226179":"2024-04-25T16:15:43.572541+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T16:20:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:03 smithi070 ceph-mon[21827]: osdmap e57: 8 total, 8 up, 8 in 2024-04-24T16:20:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:03 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1774426440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:04.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:03 smithi070 ceph-mon[26293]: osdmap e57: 8 total, 8 up, 8 in 2024-04-24T16:20:04.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:03 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1774426440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:04.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:03 smithi156 ceph-mon[22679]: osdmap e57: 8 total, 8 up, 8 in 2024-04-24T16:20:04.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:03 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1774426440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:04.347 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T16:18:11.216254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-24T16:20:04.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-24T16:20:04.555 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:04.930 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:04.923+0000 7fae81e97700 -1 calc_pg_upmaps abort due to max <= 0 2024-04-24T16:20:04.933 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:04 smithi070 ceph-mon[26293]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:04.933 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:04 smithi070 ceph-mon[26293]: osdmap e58: 8 total, 8 up, 8 in 2024-04-24T16:20:04.933 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:04 smithi070 ceph-mon[21827]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:04.933 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:04 smithi070 ceph-mon[21827]: osdmap e58: 8 total, 8 up, 8 in 2024-04-24T16:20:05.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:04 smithi156 ceph-mon[22679]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:05.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:04 smithi156 ceph-mon[22679]: osdmap e58: 8 total, 8 up, 8 in 2024-04-24T16:20:05.859 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:05 smithi070 ceph-mon[26293]: osdmap e59: 8 total, 8 up, 8 in 2024-04-24T16:20:05.859 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:05.859 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:05 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:05 smithi070 ceph-mon[21827]: osdmap e59: 8 total, 8 up, 8 in 2024-04-24T16:20:05.859 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:05.860 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:05 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:06.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:05 smithi156 ceph-mon[22679]: osdmap e59: 8 total, 8 up, 8 in 2024-04-24T16:20:06.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:06.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:05 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:06.227 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-24T16:20:06.829 INFO:tasks.cephadm:Adding prometheus.a on smithi156 2024-04-24T16:20:06.829 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch apply prometheus '1;smithi156=a' 2024-04-24T16:20:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:06 smithi070 ceph-mon[21827]: pgmap v169: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T16:20:07.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:06 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3009435193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T16:20:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:06 smithi070 ceph-mon[26293]: pgmap v169: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T16:20:07.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:06 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3009435193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T16:20:07.145 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:06 smithi156 ceph-mon[22679]: pgmap v169: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T16:20:07.145 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:06 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3009435193' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T16:20:08.703 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled prometheus update... 2024-04-24T16:20:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:08 smithi156 ceph-mon[22679]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-24T16:20:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:08 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:08 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:08 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:20:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:08 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:20:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:08 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:08 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[21827]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[26293]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:09.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:20:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:20:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' 2024-04-24T16:20:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:08 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-24T16:20:09.305 DEBUG:teuthology.orchestra.run.smithi156:prometheus.a> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service 2024-04-24T16:20:09.307 INFO:tasks.cephadm:Adding node-exporter.a on smithi070 2024-04-24T16:20:09.307 INFO:tasks.cephadm:Adding node-exporter.b on smithi156 2024-04-24T16:20:09.307 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch apply node-exporter '2;smithi070=a;smithi156=b' 2024-04-24T16:20:09.818 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ignoring --setuser ceph since I am not root 2024-04-24T16:20:09.818 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ignoring --setgroup ceph since I am not root 2024-04-24T16:20:09.818 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-mgr[23857]: -- 172.21.15.156:0/1536599125 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564472e42340 con 0x564472e1b000 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[21827]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi156=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[21827]: Saving service prometheus spec with placement smithi156=a;count:1 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[21827]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[21827]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[26293]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi156=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[26293]: Saving service prometheus spec with placement smithi156=a;count:1 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[26293]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-24T16:20:10.012 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mon[26293]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-24T16:20:10.013 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setuser ceph since I am not root 2024-04-24T16:20:10.013 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setgroup ceph since I am not root 2024-04-24T16:20:10.013 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-mgr[22061]: -- 172.21.15.70:0/3277799498 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5594bd6f6340 con 0x5594bd6cec00 2024-04-24T16:20:10.013 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:09.887+0000 7fe20702f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:20:10.013 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:10.013+0000 7fe20702f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:20:10.129 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-mon[22679]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi156=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:10.129 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-mon[22679]: Saving service prometheus spec with placement smithi156=a;count:1 2024-04-24T16:20:10.129 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-mon[22679]: from='mgr.14152 172.21.15.70:0/1728444035' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-24T16:20:10.129 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-mon[22679]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-24T16:20:10.130 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:09.891+0000 7faafd6fe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:20:10.130 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:10.031+0000 7faafd6fe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:20:10.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:10.116+0000 7fe20702f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:20:10.407 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:10.131+0000 7faafd6fe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:20:10.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:10.436+0000 7fe20702f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:20:10.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:10.522+0000 7fe20702f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:20:10.792 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:10.448+0000 7faafd6fe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:20:10.792 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:10.547+0000 7faafd6fe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:20:12.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:11 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:11.675+0000 7fe20702f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:20:12.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:11 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:11.837+0000 7fe20702f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:20:12.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:11 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:11.917+0000 7fe20702f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:20:12.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:11.680+0000 7faafd6fe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:20:12.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:11.842+0000 7faafd6fe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:20:12.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:11.922+0000 7faafd6fe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:20:12.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:12 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:12.225+0000 7fe20702f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:20:12.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:12 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:12.316+0000 7fe20702f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:20:12.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:12 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:12.406+0000 7fe20702f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:20:12.652 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:12.231+0000 7faafd6fe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:20:12.652 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:12.320+0000 7faafd6fe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:20:12.652 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:12.411+0000 7faafd6fe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:20:12.907 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:12.654+0000 7faafd6fe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:20:13.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:12 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:12.643+0000 7fe20702f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:20:13.408 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:13.093+0000 7faafd6fe000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:20:13.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:13.077+0000 7fe20702f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:20:13.672 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:13.581+0000 7faafd6fe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:20:13.672 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:13.673+0000 7faafd6fe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:20:13.937 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:13.553+0000 7fe20702f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:20:13.938 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:13.645+0000 7fe20702f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:20:13.938 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:13.936+0000 7fe20702f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:20:14.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:14.025+0000 7fe20702f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:20:14.407 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:13.963+0000 7faafd6fe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:20:14.408 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:14.051+0000 7faafd6fe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:20:14.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:14.424+0000 7fe20702f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:20:14.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:14.556+0000 7fe20702f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:20:14.758 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:14.646+0000 7fe20702f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:20:14.758 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:14.759+0000 7fe20702f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:20:14.788 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:14.456+0000 7faafd6fe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:20:14.789 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:14.586+0000 7faafd6fe000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:20:14.789 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:14.677+0000 7faafd6fe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:20:14.789 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:14.791+0000 7faafd6fe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:20:15.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:14.849+0000 7fe20702f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:20:15.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:14.879+0000 7faafd6fe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:20:15.743 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:15.392+0000 7fe20702f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:20:15.787 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:15.421+0000 7faafd6fe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: Active manager daemon y restarted 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: Activating manager daemon y 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: osdmap e60: 8 total, 8 up, 8 in 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: Standby manager daemon x restarted 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:20:16.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:15.745+0000 7fe20702f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:20:15] ENGINE Bus STARTING 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: Active manager daemon y restarted 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: Activating manager daemon y 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: osdmap e60: 8 total, 8 up, 8 in 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: Standby manager daemon x restarted 2024-04-24T16:20:16.006 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:20:16.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:20:16.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:20:16.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:20:16.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:15 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:20:16.108 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:15.790+0000 7faafd6fe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: [24/Apr/2024:16:20:15] ENGINE Bus STARTING 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: CherryPy Checker: 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: The Application mounted at '' has an empty config. 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: [24/Apr/2024:16:20:15] ENGINE Serving on http://:::9283 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: [24/Apr/2024:16:20:15] ENGINE Bus STARTED 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: Active manager daemon y restarted 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: Activating manager daemon y 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: osdmap e60: 8 total, 8 up, 8 in 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: Standby manager daemon x restarted 2024-04-24T16:20:16.109 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:20:16.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:20:16.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:20:16.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:20:16.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:15 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/1376523891' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:20:16.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: CherryPy Checker: 2024-04-24T16:20:16.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: The Application mounted at '' has an empty config. 2024-04-24T16:20:16.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:20:16.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:20:16] ENGINE Serving on http://:::9283 2024-04-24T16:20:16.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:20:16] ENGINE Bus STARTED 2024-04-24T16:20:16.740 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:20:16] ENGINE Bus STARTING 2024-04-24T16:20:16.740 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:20:16] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:20:16.740 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:20:16] ENGINE Bus STARTED 2024-04-24T16:20:16.823 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled node-exporter update... 2024-04-24T16:20:16.824 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: mgrmap e17: y(active, starting, since 0.0550544s), standbys: x 2024-04-24T16:20:16.824 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:20:16.824 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:20:16.824 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:20:16.824 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:20:16.824 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: Manager daemon y is now available 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:16.825 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: [24/Apr/2024:16:20:16] ENGINE Bus STARTING 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: [24/Apr/2024:16:20:16] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: [24/Apr/2024:16:20:16] ENGINE Bus STARTED 2024-04-24T16:20:16.826 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: mgrmap e17: y(active, starting, since 0.0550544s), standbys: x 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:20:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: Manager daemon y is now available 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: [24/Apr/2024:16:20:16] ENGINE Bus STARTING 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: [24/Apr/2024:16:20:16] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: [24/Apr/2024:16:20:16] ENGINE Bus STARTED 2024-04-24T16:20:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: mgrmap e17: y(active, starting, since 0.0550544s), standbys: x 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:20:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: Manager daemon y is now available 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:20:17.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: [24/Apr/2024:16:20:16] ENGINE Bus STARTING 2024-04-24T16:20:17.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: [24/Apr/2024:16:20:16] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:20:17.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: [24/Apr/2024:16:20:16] ENGINE Bus STARTED 2024-04-24T16:20:17.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.455 DEBUG:teuthology.orchestra.run.smithi070:node-exporter.a> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service 2024-04-24T16:20:17.458 DEBUG:teuthology.orchestra.run.smithi156:node-exporter.b> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service 2024-04-24T16:20:17.460 INFO:tasks.cephadm:Adding alertmanager.a on smithi070 2024-04-24T16:20:17.460 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch apply alertmanager '1;smithi070=a' 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[26293]: mgrmap e18: y(active, since 1.06834s), standbys: x 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[26293]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi070=a;smithi156=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[26293]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[26293]: Saving service node-exporter spec with placement smithi070=a;smithi156=b;count:2 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.836 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[21827]: mgrmap e18: y(active, since 1.06834s), standbys: x 2024-04-24T16:20:17.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[21827]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi070=a;smithi156=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:17.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[21827]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:17.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[21827]: Saving service node-exporter spec with placement smithi070=a;smithi156=b;count:2 2024-04-24T16:20:17.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:17 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.992 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:17 smithi156 ceph-mon[22679]: mgrmap e18: y(active, since 1.06834s), standbys: x 2024-04-24T16:20:17.992 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:17 smithi156 ceph-mon[22679]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi070=a;smithi156=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:17.993 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:17 smithi156 ceph-mon[22679]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:17.993 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:17 smithi156 ceph-mon[22679]: Saving service node-exporter spec with placement smithi070=a;smithi156=b;count:2 2024-04-24T16:20:17.993 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:17 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.993 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:17 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:17.993 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:17 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:18.853 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:18 smithi156 ceph-mon[22679]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:18.854 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:18 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:18.854 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:18 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:20:18.854 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:18 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:20:18.854 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:18 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:18.854 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:18 smithi156 ceph-mon[22679]: mgrmap e19: y(active, since 3s), standbys: x 2024-04-24T16:20:19.255 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled alertmanager update... 2024-04-24T16:20:19.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[26293]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:19.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[26293]: mgrmap e19: y(active, since 3s), standbys: x 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[21827]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:18 smithi070 ceph-mon[21827]: mgrmap e19: y(active, since 3s), standbys: x 2024-04-24T16:20:19.944 DEBUG:teuthology.orchestra.run.smithi070:alertmanager.a> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service 2024-04-24T16:20:19.947 INFO:tasks.cephadm:Adding grafana.a on smithi156 2024-04-24T16:20:19.947 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph orch apply grafana '1;smithi156=a' 2024-04-24T16:20:20.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi070=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:20.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: Saving service alertmanager spec with placement smithi070=a;count:1 2024-04-24T16:20:20.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:20:20.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:20:20.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi070=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:20.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: Saving service alertmanager spec with placement smithi070=a;count:1 2024-04-24T16:20:20.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:20:20.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:20:20.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:20 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.548 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi070=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:20.549 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: Saving service alertmanager spec with placement smithi070=a;count:1 2024-04-24T16:20:20.549 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.549 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.549 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:20:20.549 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:20:20.549 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:20.549 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:20 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:21.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:21 smithi070 ceph-mon[26293]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:21.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:21 smithi070 ceph-mon[26293]: Deploying daemon prometheus.a on smithi156 2024-04-24T16:20:21.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:21 smithi070 ceph-mon[21827]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:21.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:21 smithi070 ceph-mon[21827]: Deploying daemon prometheus.a on smithi156 2024-04-24T16:20:21.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:21 smithi156 ceph-mon[22679]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:21.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:21 smithi156 ceph-mon[22679]: Deploying daemon prometheus.a on smithi156 2024-04-24T16:20:22.292 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled grafana update... 2024-04-24T16:20:23.330 DEBUG:teuthology.orchestra.run.smithi156:grafana.a> sudo journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@grafana.a.service 2024-04-24T16:20:23.333 INFO:tasks.cephadm:Setting up client nodes... 2024-04-24T16:20:23.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-24T16:20:23.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[26293]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:23.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[26293]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi156=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:23.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[26293]: Saving service grafana spec with placement smithi156=a;count:1 2024-04-24T16:20:23.634 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:23.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[21827]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:23.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[21827]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi156=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:23.634 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[21827]: Saving service grafana spec with placement smithi156=a;count:1 2024-04-24T16:20:23.635 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:23 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:23.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:23 smithi156 ceph-mon[22679]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:23.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:23 smithi156 ceph-mon[22679]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi156=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:20:23.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:23 smithi156 ceph-mon[22679]: Saving service grafana spec with placement smithi156=a;count:1 2024-04-24T16:20:23.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:23 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:25.351 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-24T16:20:25.351 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBJMSlmvFHvFBAAqLozD6dfkjwDLh+gixeE1w== 2024-04-24T16:20:25.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:25 smithi156 ceph-mon[22679]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:25.662 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:25 smithi070 ceph-mon[26293]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:25.662 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:25 smithi070 ceph-mon[21827]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:26.088 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-24T16:20:26.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-24T16:20:26.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T16:20:26.125 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-24T16:20:26.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:26 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4258479130' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:26.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:26 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4258479130' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T16:20:26.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:26 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4258479130' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:26.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:26 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4258479130' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T16:20:26.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:26 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4258479130' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:26.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:26 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4258479130' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T16:20:27.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:27 smithi156 ceph-mon[22679]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:27.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:27 smithi070 ceph-mon[26293]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:27.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:27 smithi070 ceph-mon[21827]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:29.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:29 smithi156 ceph-mon[22679]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:30.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:29 smithi070 ceph-mon[26293]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:30.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:29 smithi070 ceph-mon[21827]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:30.795 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2024-04-24T16:20:30.795 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBOMSlmPs5JLxAAhqRf+EYfufYgUo1eL967vw== 2024-04-24T16:20:31.819 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:31 smithi156 ceph-mon[22679]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:31.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:31 smithi156 ceph-mon[22679]: from='client.? 172.21.15.156:0/3355231914' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:31.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:31 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:31.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:31 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T16:20:31.842 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T16:20:31.842 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-24T16:20:31.842 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-24T16:20:31.882 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T16:20:31.882 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-24T16:20:31.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-24T16:20:31.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[21827]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:31.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[21827]: from='client.? 172.21.15.156:0/3355231914' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:31.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:31.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T16:20:31.907 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[26293]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:31.907 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[26293]: from='client.? 172.21.15.156:0/3355231914' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:31.907 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T16:20:31.908 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:31 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T16:20:32.100 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:33.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:33 smithi070 ceph-mon[26293]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:33.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:33 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:33.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:33 smithi070 ceph-mon[26293]: Deploying daemon node-exporter.a on smithi070 2024-04-24T16:20:33.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:33 smithi070 ceph-mon[21827]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:33.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:33 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:33.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:33 smithi070 ceph-mon[21827]: Deploying daemon node-exporter.a on smithi070 2024-04-24T16:20:33.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:33 smithi156 ceph-mon[22679]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:33.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:33 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:33.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:33 smithi156 ceph-mon[22679]: Deploying daemon node-exporter.a on smithi070 2024-04-24T16:20:34.045 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:20:34.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:34 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/39472568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:20:34.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:34 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/39472568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:20:34.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:34 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/39472568' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:20:34.695 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":19,"active_gid":24328,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2453531589},{"type":"v1","addr":"172.21.15.70:6801","nonce":2453531589}]},"active_addr":"172.21.15.70:6801/2453531589","active_change":"2024-04-24T16:20:15.748952+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34319,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.70:8443/","prometheus":"http://172.21.15.70:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2234875754}]},{"addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2241868720}]},{"addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3427395814}]},{"addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3930276067}]}]}} 2024-04-24T16:20:34.700 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-24T16:20:34.700 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-24T16:20:34.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T16:20:34.992 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:35.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:35 smithi070 ceph-mon[26293]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:35.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:35 smithi070 ceph-mon[21827]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:35.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:35 smithi156 ceph-mon[22679]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:36 smithi156 ceph-mon[22679]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:37.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:36 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:36 smithi070 ceph-mon[21827]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:36 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:36 smithi070 ceph-mon[26293]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:36 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:37.638 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:37 smithi070 bash[41127]: Writing manifest to image destination 2024-04-24T16:20:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:20:37.833 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":60,"fsid":"be31be9a-0255-11ef-bc93-c7b262605968","created":"2024-04-24T16:15:17.566184+0000","modified":"2024-04-24T16:20:15.748610+0000","last_up_change":"2024-04-24T16:20:00.786057+0000","last_in_change":"2024-04-24T16:19:44.833161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T16:18:11.216254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1479dc88-7964-4663-a0e4-7c385281030b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6803","nonce":351334718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6805","nonce":351334718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6809","nonce":351334718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6807","nonce":351334718}]},"public_addr":"172.21.15.70:6803/351334718","cluster_addr":"172.21.15.70:6805/351334718","heartbeat_back_addr":"172.21.15.70:6809/351334718","heartbeat_front_addr":"172.21.15.70:6807/351334718","state":["exists","up"]},{"osd":1,"uuid":"48295f9b-7ffc-4b68-99b6-7599af8a0aff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6811","nonce":2849413144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6813","nonce":2849413144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6817","nonce":2849413144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6815","nonce":2849413144}]},"public_addr":"172.21.15.70:6811/2849413144","cluster_addr":"172.21.15.70:6813/2849413144","heartbeat_back_addr":"172.21.15.70:6817/2849413144","heartbeat_front_addr":"172.21.15.70:6815/2849413144","state":["exists","up"]},{"osd":2,"uuid":"22be76f9-af84-4147-9dd5-99e47e3b3ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6819","nonce":679611873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6821","nonce":679611873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6825","nonce":679611873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6823","nonce":679611873}]},"public_addr":"172.21.15.70:6819/679611873","cluster_addr":"172.21.15.70:6821/679611873","heartbeat_back_addr":"172.21.15.70:6825/679611873","heartbeat_front_addr":"172.21.15.70:6823/679611873","state":["exists","up"]},{"osd":3,"uuid":"f2f8f93a-5321-4567-9551-cd2b1aa28691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6827","nonce":1561941519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6829","nonce":1561941519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6833","nonce":1561941519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6831","nonce":1561941519}]},"public_addr":"172.21.15.70:6827/1561941519","cluster_addr":"172.21.15.70:6829/1561941519","heartbeat_back_addr":"172.21.15.70:6833/1561941519","heartbeat_front_addr":"172.21.15.70:6831/1561941519","state":["exists","up"]},{"osd":4,"uuid":"75d4ec4e-da7f-43d7-87e4-72ab045af699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6801","nonce":1959136868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6803","nonce":1959136868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6807","nonce":1959136868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6805","nonce":1959136868}]},"public_addr":"172.21.15.156:6801/1959136868","cluster_addr":"172.21.15.156:6803/1959136868","heartbeat_back_addr":"172.21.15.156:6807/1959136868","heartbeat_front_addr":"172.21.15.156:6805/1959136868","state":["exists","up"]},{"osd":5,"uuid":"ccc7fa7c-53c3-46ae-af1f-44c1bee32330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6809","nonce":4243847227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6811","nonce":4243847227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6815","nonce":4243847227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6813","nonce":4243847227}]},"public_addr":"172.21.15.156:6809/4243847227","cluster_addr":"172.21.15.156:6811/4243847227","heartbeat_back_addr":"172.21.15.156:6815/4243847227","heartbeat_front_addr":"172.21.15.156:6813/4243847227","state":["exists","up"]},{"osd":6,"uuid":"5d423492-ffef-466d-afa8-ea869cf99ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6817","nonce":1319206544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6819","nonce":1319206544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6823","nonce":1319206544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6821","nonce":1319206544}]},"public_addr":"172.21.15.156:6817/1319206544","cluster_addr":"172.21.15.156:6819/1319206544","heartbeat_back_addr":"172.21.15.156:6823/1319206544","heartbeat_front_addr":"172.21.15.156:6821/1319206544","state":["exists","up"]},{"osd":7,"uuid":"0bad0ad0-a88c-4f1c-8010-35b7fbabe791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6825","nonce":1364928815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6827","nonce":1364928815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6831","nonce":1364928815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6829","nonce":1364928815}]},"public_addr":"172.21.15.156:6825/1364928815","cluster_addr":"172.21.15.156:6827/1364928815","heartbeat_back_addr":"172.21.15.156:6831/1364928815","heartbeat_front_addr":"172.21.15.156:6829/1364928815","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:17:20.883636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:17:43.512527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:07.126511+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:30.889304+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:51.518774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:14.757678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:36.844389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:59.520858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/2767731331":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/4043624147":"2024-04-25T16:16:04.858290+0000","172.21.15.70:6800/2548077742":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/2490199536":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/4129610450":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/1502360754":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/1351576920":"2024-04-25T16:15:43.572541+0000","172.21.15.70:0/2897538162":"2024-04-25T16:15:43.572541+0000","172.21.15.70:6800/3814823340":"2024-04-25T16:15:43.572541+0000","172.21.15.70:0/752255822":"2024-04-25T16:20:15.748568+0000","172.21.15.70:6801/2548077742":"2024-04-25T16:16:04.858290+0000","172.21.15.70:6801/3814823340":"2024-04-25T16:15:43.572541+0000","172.21.15.70:6801/2944003385":"2024-04-25T16:20:15.748568+0000","172.21.15.70:6800/2944003385":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/312180501":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/2369226179":"2024-04-25T16:15:43.572541+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T16:20:38.335 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 podman[41127]: 2024-04-24 16:20:38.030060329 +0000 UTC m=+3.245106695 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-24T16:20:38.335 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 podman[41127]: 2024-04-24 16:20:38.236958219 +0000 UTC m=+3.452004572 container create 57518b11b60ba9590825cb5d13e502d2ed30dd34374e1acde851176c5319cef2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:20:38.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4151991454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:38.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4151991454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:38.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:38.791 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 podman[41127]: 2024-04-24 16:20:38.422115 +0000 UTC m=+3.637161358 container init 57518b11b60ba9590825cb5d13e502d2ed30dd34374e1acde851176c5319cef2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 podman[41127]: 2024-04-24 16:20:38.424972048 +0000 UTC m=+3.640018403 container start 57518b11b60ba9590825cb5d13e502d2ed30dd34374e1acde851176c5319cef2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.429Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.429Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.429Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.429Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=arp 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-24T16:20:38.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=edac 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-24T16:20:38.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=os 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=stat 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-24T16:20:38.794 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=time 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=uname 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.430Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[41302]: ts=2024-04-24T16:20:38.431Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 bash[41127]: 57518b11b60ba9590825cb5d13e502d2ed30dd34374e1acde851176c5319cef2 2024-04-24T16:20:38.795 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:20:38 smithi070 systemd[1]: Started Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:20:38.848 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:38 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4151991454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:38.944 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:38 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:39.038 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-24T16:20:39.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T16:20:39.251 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:39.579 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:20:39] "GET /metrics HTTP/1.1" 200 191520 "" "Prometheus/2.33.4" 2024-04-24T16:20:39.596 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:39 smithi156 bash[38032]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-24T16:20:39.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:39 smithi156 ceph-mon[22679]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:39.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:39 smithi156 ceph-mon[22679]: Deploying daemon node-exporter.b on smithi156 2024-04-24T16:20:39.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:39 smithi156 bash[38032]: Getting image source signatures 2024-04-24T16:20:39.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:39 smithi156 bash[38032]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-24T16:20:39.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:39 smithi156 bash[38032]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-24T16:20:39.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:39 smithi156 bash[38032]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-24T16:20:40.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:39 smithi070 ceph-mon[26293]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:40.004 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:39 smithi070 ceph-mon[26293]: Deploying daemon node-exporter.b on smithi156 2024-04-24T16:20:40.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:39 smithi070 ceph-mon[21827]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:40.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:39 smithi070 ceph-mon[21827]: Deploying daemon node-exporter.b on smithi156 2024-04-24T16:20:41.128 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:20:41.128 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":60,"fsid":"be31be9a-0255-11ef-bc93-c7b262605968","created":"2024-04-24T16:15:17.566184+0000","modified":"2024-04-24T16:20:15.748610+0000","last_up_change":"2024-04-24T16:20:00.786057+0000","last_in_change":"2024-04-24T16:19:44.833161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T16:18:11.216254+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1479dc88-7964-4663-a0e4-7c385281030b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6803","nonce":351334718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6805","nonce":351334718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6809","nonce":351334718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":351334718},{"type":"v1","addr":"172.21.15.70:6807","nonce":351334718}]},"public_addr":"172.21.15.70:6803/351334718","cluster_addr":"172.21.15.70:6805/351334718","heartbeat_back_addr":"172.21.15.70:6809/351334718","heartbeat_front_addr":"172.21.15.70:6807/351334718","state":["exists","up"]},{"osd":1,"uuid":"48295f9b-7ffc-4b68-99b6-7599af8a0aff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6811","nonce":2849413144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6813","nonce":2849413144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6817","nonce":2849413144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2849413144},{"type":"v1","addr":"172.21.15.70:6815","nonce":2849413144}]},"public_addr":"172.21.15.70:6811/2849413144","cluster_addr":"172.21.15.70:6813/2849413144","heartbeat_back_addr":"172.21.15.70:6817/2849413144","heartbeat_front_addr":"172.21.15.70:6815/2849413144","state":["exists","up"]},{"osd":2,"uuid":"22be76f9-af84-4147-9dd5-99e47e3b3ea4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6819","nonce":679611873}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6821","nonce":679611873}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6825","nonce":679611873}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":679611873},{"type":"v1","addr":"172.21.15.70:6823","nonce":679611873}]},"public_addr":"172.21.15.70:6819/679611873","cluster_addr":"172.21.15.70:6821/679611873","heartbeat_back_addr":"172.21.15.70:6825/679611873","heartbeat_front_addr":"172.21.15.70:6823/679611873","state":["exists","up"]},{"osd":3,"uuid":"f2f8f93a-5321-4567-9551-cd2b1aa28691","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6827","nonce":1561941519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6829","nonce":1561941519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6833","nonce":1561941519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1561941519},{"type":"v1","addr":"172.21.15.70:6831","nonce":1561941519}]},"public_addr":"172.21.15.70:6827/1561941519","cluster_addr":"172.21.15.70:6829/1561941519","heartbeat_back_addr":"172.21.15.70:6833/1561941519","heartbeat_front_addr":"172.21.15.70:6831/1561941519","state":["exists","up"]},{"osd":4,"uuid":"75d4ec4e-da7f-43d7-87e4-72ab045af699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6801","nonce":1959136868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6803","nonce":1959136868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6807","nonce":1959136868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1959136868},{"type":"v1","addr":"172.21.15.156:6805","nonce":1959136868}]},"public_addr":"172.21.15.156:6801/1959136868","cluster_addr":"172.21.15.156:6803/1959136868","heartbeat_back_addr":"172.21.15.156:6807/1959136868","heartbeat_front_addr":"172.21.15.156:6805/1959136868","state":["exists","up"]},{"osd":5,"uuid":"ccc7fa7c-53c3-46ae-af1f-44c1bee32330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6809","nonce":4243847227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6811","nonce":4243847227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6815","nonce":4243847227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":4243847227},{"type":"v1","addr":"172.21.15.156:6813","nonce":4243847227}]},"public_addr":"172.21.15.156:6809/4243847227","cluster_addr":"172.21.15.156:6811/4243847227","heartbeat_back_addr":"172.21.15.156:6815/4243847227","heartbeat_front_addr":"172.21.15.156:6813/4243847227","state":["exists","up"]},{"osd":6,"uuid":"5d423492-ffef-466d-afa8-ea869cf99ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6817","nonce":1319206544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6819","nonce":1319206544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6823","nonce":1319206544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1319206544},{"type":"v1","addr":"172.21.15.156:6821","nonce":1319206544}]},"public_addr":"172.21.15.156:6817/1319206544","cluster_addr":"172.21.15.156:6819/1319206544","heartbeat_back_addr":"172.21.15.156:6823/1319206544","heartbeat_front_addr":"172.21.15.156:6821/1319206544","state":["exists","up"]},{"osd":7,"uuid":"0bad0ad0-a88c-4f1c-8010-35b7fbabe791","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6825","nonce":1364928815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6827","nonce":1364928815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6831","nonce":1364928815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1364928815},{"type":"v1","addr":"172.21.15.156:6829","nonce":1364928815}]},"public_addr":"172.21.15.156:6825/1364928815","cluster_addr":"172.21.15.156:6827/1364928815","heartbeat_back_addr":"172.21.15.156:6831/1364928815","heartbeat_front_addr":"172.21.15.156:6829/1364928815","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:17:20.883636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:17:43.512527+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:07.126511+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:30.889304+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:18:51.518774+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:14.757678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:36.844389+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T16:19:59.520858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/2767731331":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/4043624147":"2024-04-25T16:16:04.858290+0000","172.21.15.70:6800/2548077742":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/2490199536":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/4129610450":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/1502360754":"2024-04-25T16:16:04.858290+0000","172.21.15.70:0/1351576920":"2024-04-25T16:15:43.572541+0000","172.21.15.70:0/2897538162":"2024-04-25T16:15:43.572541+0000","172.21.15.70:6800/3814823340":"2024-04-25T16:15:43.572541+0000","172.21.15.70:0/752255822":"2024-04-25T16:20:15.748568+0000","172.21.15.70:6801/2548077742":"2024-04-25T16:16:04.858290+0000","172.21.15.70:6801/3814823340":"2024-04-25T16:15:43.572541+0000","172.21.15.70:6801/2944003385":"2024-04-25T16:20:15.748568+0000","172.21.15.70:6800/2944003385":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/312180501":"2024-04-25T16:20:15.748568+0000","172.21.15.70:0/2369226179":"2024-04-25T16:15:43.572541+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T16:20:41.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:41 smithi070 ceph-mon[26293]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:41.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:41 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3647038380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:41.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:41 smithi070 ceph-mon[21827]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:41.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:41 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3647038380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:41.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-24T16:20:41.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-24T16:20:41.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-24T16:20:41.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-24T16:20:41.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-24T16:20:41.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-24T16:20:41.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-24T16:20:41.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-24T16:20:41.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:41 smithi156 bash[38032]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-24T16:20:41.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:41 smithi156 bash[38032]: Writing manifest to image destination 2024-04-24T16:20:41.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:41 smithi156 ceph-mon[22679]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:41.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:41 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3647038380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T16:20:42.199 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.201 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.201 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.202 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.203 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.207 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.207 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:42.408 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 podman[38032]: 2024-04-24 16:20:42.053694753 +0000 UTC m=+2.740452475 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 podman[38032]: 2024-04-24 16:20:42.144008628 +0000 UTC m=+2.830766337 container create a60500a596252b835a3f352020a68eee6d03bb718b960c3d3fbc3378dc09314c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 podman[38032]: 2024-04-24 16:20:42.337651935 +0000 UTC m=+3.024409644 container init a60500a596252b835a3f352020a68eee6d03bb718b960c3d3fbc3378dc09314c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 podman[38032]: 2024-04-24 16:20:42.340235034 +0000 UTC m=+3.026992746 container start a60500a596252b835a3f352020a68eee6d03bb718b960c3d3fbc3378dc09314c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.343Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.343Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.344Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.344Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.344Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.344Z caller=node_exporter.go:115 level=info collector=arp 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.344Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-24T16:20:42.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=edac 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-24T16:20:42.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=os 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-24T16:20:42.411 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=stat 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=time 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=uname 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-24T16:20:42.412 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[38083]: ts=2024-04-24T16:20:42.345Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-24T16:20:42.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:20:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:20:42.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 bash[38032]: a60500a596252b835a3f352020a68eee6d03bb718b960c3d3fbc3378dc09314c 2024-04-24T16:20:42.908 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:20:42 smithi156 systemd[1]: Started Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:20:43.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:43 smithi070 ceph-mon[26293]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:43.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:43 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:43.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:43 smithi070 ceph-mon[26293]: Deploying daemon alertmanager.a on smithi070 2024-04-24T16:20:43.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:43 smithi070 ceph-mon[21827]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:43.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:43 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:43.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:43 smithi070 ceph-mon[21827]: Deploying daemon alertmanager.a on smithi070 2024-04-24T16:20:43.809 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:43 smithi156 ceph-mon[22679]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:43.810 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:43 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:43.810 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:43 smithi156 ceph-mon[22679]: Deploying daemon alertmanager.a on smithi070 2024-04-24T16:20:45.669 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:45 smithi070 ceph-mon[26293]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:45.669 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:45 smithi070 ceph-mon[21827]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:45 smithi156 ceph-mon[22679]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:47.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:46 smithi070 ceph-mon[26293]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:47.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:46 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:47.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:46 smithi070 ceph-mon[21827]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:47.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:46 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:47.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:46 smithi156 ceph-mon[22679]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:47.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:46 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:20:49.595 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:20:49] "GET /metrics HTTP/1.1" 200 191493 "" "Prometheus/2.33.4" 2024-04-24T16:20:49.738 INFO:teuthology.orchestra.run.smithi070.stdout:133143986201 2024-04-24T16:20:49.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-24T16:20:49.855 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:49 smithi070 ceph-mon[26293]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:49.856 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:49 smithi070 ceph-mon[21827]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:49.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:49 smithi156 ceph-mon[22679]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:51.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:51 smithi070 ceph-mon[26293]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:51.796 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:51 smithi070 ceph-mon[21827]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:51.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:51 smithi156 ceph-mon[22679]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:52.460 INFO:teuthology.orchestra.run.smithi070.stdout:236223201292 2024-04-24T16:20:52.461 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-24T16:20:53.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:20:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:20:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:20:53.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:53 smithi156 ceph-mon[22679]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:54.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:53 smithi070 ceph-mon[21827]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:54.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:53 smithi070 ceph-mon[26293]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:55.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:55 smithi070 ceph-mon[21827]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:55.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:55 smithi070 ceph-mon[26293]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:55.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:55 smithi156 ceph-mon[22679]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:57.707 INFO:teuthology.orchestra.run.smithi070.stdout:34359738413 2024-04-24T16:20:57.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-24T16:20:57.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:57 smithi156 ceph-mon[22679]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:57.968 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:57 smithi070 ceph-mon[26293]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:57.968 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:57 smithi070 ceph-mon[21827]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:201863462929 2024-04-24T16:20:58.124 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-24T16:20:58.458 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:58.479 INFO:teuthology.orchestra.run.smithi070.stdout:55834574886 2024-04-24T16:20:58.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-24T16:20:58.689 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:20:59.470 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:20:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:20:59] "GET /metrics HTTP/1.1" 200 191493 "" "Prometheus/2.33.4" 2024-04-24T16:20:59.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:20:59 smithi070 ceph-mon[26293]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:59.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:20:59 smithi070 ceph-mon[21827]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:20:59.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:20:59 smithi156 ceph-mon[22679]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:01.125 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:01.155 INFO:teuthology.orchestra.run.smithi070.stdout:111669149727 2024-04-24T16:21:01.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-24T16:21:01.509 INFO:teuthology.orchestra.run.smithi070.stdout:77309411363 2024-04-24T16:21:01.510 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-24T16:21:01.758 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:01 smithi070 ceph-mon[26293]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:01.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:01 smithi070 ceph-mon[21827]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:01.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:01 smithi156 ceph-mon[22679]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:02.129 INFO:teuthology.orchestra.run.smithi070.stdout:167503724566 2024-04-24T16:21:02.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-24T16:21:02.148 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:03.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:21:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:21:03.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:03 smithi070 ceph-mon[26293]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:03.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:03 smithi070 ceph-mon[21827]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:03.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:03 smithi156 ceph-mon[22679]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:05.187 INFO:teuthology.orchestra.run.smithi070.stdout:133143986204 2024-04-24T16:21:05.516 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:05.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:05 smithi070 ceph-mon[21827]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:05.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:05 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/752605759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T16:21:05.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:05 smithi070 ceph-mon[26293]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:05.683 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:05 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/752605759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T16:21:05.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:05 smithi156 ceph-mon[22679]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:05.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:05 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/752605759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T16:21:06.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986201 got 133143986204 for osd.4 2024-04-24T16:21:06.505 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:07.153 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:07.153 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:07.839 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:07 smithi070 ceph-mon[26293]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:07.840 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:07 smithi070 ceph-mon[21827]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:07.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:07 smithi156 ceph-mon[22679]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:08.467 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:09.588 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:21:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:09] "GET /metrics HTTP/1.1" 200 191483 "" "Prometheus/2.33.4" 2024-04-24T16:21:09.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:09 smithi156 ceph-mon[22679]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:09.912 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:09 smithi070 ceph-mon[26293]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:09.912 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:09 smithi070 systemd[1]: Starting Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:21:09.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:09 smithi070 ceph-mon[21827]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:10.576 INFO:teuthology.orchestra.run.smithi070.stdout:34359738416 2024-04-24T16:21:10.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:10 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/385262036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T16:21:10.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:10 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/385262036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T16:21:10.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:10 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/385262036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T16:21:11.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:11 smithi070 ceph-mon[26293]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:11.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:11 smithi070 ceph-mon[21827]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:11.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:11 smithi156 ceph-mon[22679]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:11.952 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738416 for osd.0 2024-04-24T16:21:11.952 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:12.745 INFO:teuthology.orchestra.run.smithi070.stdout:55834574890 2024-04-24T16:21:13.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:21:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:21:13.361 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574886 got 55834574890 for osd.1 2024-04-24T16:21:13.362 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:13.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:13 smithi156 ceph-mon[22679]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:13.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:13 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1569609376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T16:21:14.036 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:13 smithi070 ceph-mon[26293]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:14.036 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:13 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1569609376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T16:21:14.037 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:13 smithi070 podman[43762]: 2024-04-24 16:21:13.632917412 +0000 UTC m=+0.026531189 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-24T16:21:14.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:13 smithi070 ceph-mon[21827]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:14.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:13 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1569609376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T16:21:14.418 INFO:teuthology.orchestra.run.smithi070.stdout:111669149729 2024-04-24T16:21:14.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:14 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/757377410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T16:21:14.951 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:14 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/757377410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T16:21:14.951 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:14 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/757377410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T16:21:15.047 INFO:teuthology.orchestra.run.smithi070.stdout:236223201297 2024-04-24T16:21:15.406 INFO:teuthology.orchestra.run.smithi070.stdout:77309411367 2024-04-24T16:21:15.874 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201292 got 236223201297 for osd.7 2024-04-24T16:21:15.874 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:15 smithi156 ceph-mon[22679]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:15 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1984002161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T16:21:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:15 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1865783037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T16:21:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:15 smithi070 ceph-mon[21827]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:15 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1984002161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T16:21:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:15 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1865783037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T16:21:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:15 smithi070 ceph-mon[26293]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:15 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1984002161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T16:21:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:15 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1865783037' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T16:21:16.671 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149729 for osd.3 2024-04-24T16:21:16.671 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:16.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:21:16.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:21:16.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:21:16.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:21:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:21:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:21:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:21:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:21:17.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:16 smithi070 podman[43762]: 2024-04-24 16:21:16.780692253 +0000 UTC m=+3.174306063 volume create d197124a53f76eaa5486a764d39a15c64bef3b5be4b33ab82c38a72f26e5bf60 2024-04-24T16:21:17.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:16 smithi070 podman[43762]: 2024-04-24 16:21:16.864479536 +0000 UTC m=+3.258093303 container create 4148de6cdaf1d83c2e1f63d0ab2b882c85f4fa0b4c7a512275df12191d1cca3b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:17.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:21:17.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:21:17.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:21:17.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:21:17.508 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411367 for osd.2 2024-04-24T16:21:17.508 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:17.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:17 smithi156 ceph-mon[22679]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-mon[26293]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-mon[21827]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:18.041 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 podman[43762]: 2024-04-24 16:21:17.586852591 +0000 UTC m=+3.980466359 container init 4148de6cdaf1d83c2e1f63d0ab2b882c85f4fa0b4c7a512275df12191d1cca3b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:18.041 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 podman[43762]: 2024-04-24 16:21:17.589720976 +0000 UTC m=+3.983334745 container start 4148de6cdaf1d83c2e1f63d0ab2b882c85f4fa0b4c7a512275df12191d1cca3b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:18.041 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.601Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.601Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.603Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.70 port=9094 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.605Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.630Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.630Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.633Z caller=main.go:518 msg=Listening address=:9093 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:17.633Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 bash[43762]: 4148de6cdaf1d83c2e1f63d0ab2b882c85f4fa0b4c7a512275df12191d1cca3b 2024-04-24T16:21:18.042 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:17 smithi070 systemd[1]: Started Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:21:18.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:18.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:18.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:18.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:18.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:18.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:18.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:18.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:18 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:18.939 INFO:teuthology.orchestra.run.smithi070.stdout:167503724570 2024-04-24T16:21:19.089 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:18 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:19.089 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:18 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:19.090 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:18 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:19.090 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:18 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:19.481 INFO:teuthology.orchestra.run.smithi070.stdout:201863462934 2024-04-24T16:21:19.482 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:19] "GET /metrics HTTP/1.1" 200 191494 "" "Prometheus/2.33.4" 2024-04-24T16:21:19.745 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:19.606Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000945998s 2024-04-24T16:21:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-mon[26293]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/451869576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T16:21:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/226714970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T16:21:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-mon[21827]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/451869576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T16:21:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:19 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/226714970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T16:21:20.051 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:19 smithi156 ceph-mon[22679]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:20.051 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:19 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/451869576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T16:21:20.051 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:19 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/226714970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T16:21:20.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462929 got 201863462934 for osd.6 2024-04-24T16:21:20.633 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:21.026 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724566 got 167503724570 for osd.5 2024-04-24T16:21:21.026 DEBUG:teuthology.parallel:result is None 2024-04-24T16:21:21.027 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-24T16:21:21.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T16:21:21.473 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:21.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:21 smithi070 ceph-mon[21827]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:21.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:21 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:21.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:21 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:21.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:21 smithi070 ceph-mon[26293]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:21.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:21 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:21.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:21 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:21.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:21 smithi156 ceph-mon[22679]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:21.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:21 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:21.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:21 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:23.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:21:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:21:23.532 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:21:23.534 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-24T16:21:23.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:23 smithi070 ceph-mon[26293]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:23.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:23 smithi070 ceph-mon[21827]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:23 smithi156 ceph-mon[22679]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:24.324 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-24T16:21:21.820052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48956,"kb_used_data":6140,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912260,"statfs":{"total":767960285184,"available":767910154240,"internally_reserved":0,"allocated":6287360,"data_stored":3510151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001904"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-24T16:20:15.879433+0000","last_change":"2024-04-24T16:20:04.809027+0000","last_active":"2024-04-24T16:20:15.879433+0000","last_peered":"2024-04-24T16:20:15.879433+0000","last_clean":"2024-04-24T16:20:15.879433+0000","last_became_active":"2024-04-24T16:20:04.808494+0000","last_became_peered":"2024-04-24T16:20:04.808494+0000","last_unstale":"2024-04-24T16:20:15.879433+0000","last_undegraded":"2024-04-24T16:20:15.879433+0000","last_fullsized":"2024-04-24T16:20:15.879433+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T16:18:11.266384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T16:18:11.266384+0000","last_clean_scrub_stamp":"2024-04-24T16:18:11.266384+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:27:16.795622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.58299999999999996}]},{"osd":1,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.626}]},{"osd":2,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.64300000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.53600000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.40699999999999997}]},{"osd":5,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.37},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.45100000000000001}]},{"osd":6,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.437},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52000000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.436},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.57499999999999996}]},{"osd":2,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51200000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.58799999999999997}]},{"osd":5,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48699999999999999}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.622},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.63500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.33800000000000002,"15min":0.313},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.432},{"interface":"front","average":{"1min":0.438,"5min":0.36099999999999999,"15min":0.33100000000000002},"min":{"1min":0.26700000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:21:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.376},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.38},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.46500000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.42299999999999999,"15min":0.41199999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.432,"15min":0.42399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.47599999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.753,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.60899999999999999}]},{"osd":5,"last update":"Wed Apr 24 16:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.66600000000000004}]},{"osd":6,"last update":"Wed Apr 24 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57499999999999996}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.63},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.55400000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.33500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.379},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed Apr 24 16:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.376,"15min":0.36799999999999999},"min":{"1min":0.29899999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.38500000000000001,"15min":0.38},"min":{"1min":0.29999999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.29999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64000000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.432,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.41499999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79700000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86299999999999999,"5min":0.876,"15min":0.876},"last":0.69099999999999995}]},{"osd":5,"last update":"Wed Apr 24 16:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.73599999999999999}]},{"osd":6,"last update":"Wed Apr 24 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.77700000000000002}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.71599999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:21:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.34100000000000003,"15min":0.33600000000000002},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.35999999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.312}]},{"osd":1,"last update":"Wed Apr 24 16:21:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55300000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.38300000000000001,"15min":0.38},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52900000000000003,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.36299999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53300000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46600000000000003,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.51300000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72499999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.37,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.61799999999999999}]},{"osd":5,"last update":"Wed Apr 24 16:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52600000000000002}]},{"osd":6,"last update":"Wed Apr 24 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Apr 24 16:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.498},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.64900000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41599999999999998,"15min":0.41999999999999998},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.20399999999999999}]},{"osd":1,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.59199999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.26800000000000002}]},{"osd":4,"last update":"Wed Apr 24 16:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88600000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.62,"15min":0.63},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54200000000000004}]},{"osd":5,"last update":"Wed Apr 24 16:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.45500000000000002}]},{"osd":6,"last update":"Wed Apr 24 16:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.502},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.56899999999999995}]},{"osd":7,"last update":"Wed Apr 24 16:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56200000000000006}]}]},{"osd":4,"up_from":31,"seq":133143986207,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69499999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.69399999999999995}]},{"osd":1,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.36799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.59299999999999997}]},{"osd":2,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.71199999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.71499999999999997}]},{"osd":3,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.47699999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.42299999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65200000000000002}]},{"osd":5,"last update":"Wed Apr 24 16:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55300000000000005}]},{"osd":6,"last update":"Wed Apr 24 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.495},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.67700000000000005}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.626},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.53900000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":239865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.68700000000000006}]},{"osd":1,"last update":"Wed Apr 24 16:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.57999999999999996}]},{"osd":2,"last update":"Wed Apr 24 16:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47399999999999998}]},{"osd":3,"last update":"Wed Apr 24 16:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.5},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.54600000000000004}]},{"osd":4,"last update":"Wed Apr 24 16:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.56499999999999995}]},{"osd":6,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.629}]},{"osd":7,"last update":"Wed Apr 24 16:21:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.65800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:21:24.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T16:21:24.604 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:24.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:24 smithi070 ceph-mon[26293]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T16:21:24.867 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:24 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:24.868 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:24 smithi070 ceph-mon[21827]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T16:21:24.868 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:24 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:24.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:24 smithi156 ceph-mon[22679]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T16:21:24.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:24 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[26293]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:25.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:21:25.847 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.847 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[21827]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:25.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:21:25.848 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:25 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:25 smithi156 ceph-mon[22679]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:25.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:25 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:25 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:25 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:25.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:25 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:21:25.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:25 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:26.244 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:21:26.245 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-24T16:21:26.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:26 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:21:26.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:26 smithi070 ceph-mon[21827]: Deploying daemon grafana.a on smithi156 2024-04-24T16:21:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:26 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:21:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:26 smithi070 ceph-mon[26293]: Deploying daemon grafana.a on smithi156 2024-04-24T16:21:26.797 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-24T16:21:25.820769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48956,"kb_used_data":6140,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912260,"statfs":{"total":767960285184,"available":767910154240,"internally_reserved":0,"allocated":6287360,"data_stored":3510151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002124"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-24T16:20:15.879433+0000","last_change":"2024-04-24T16:20:04.809027+0000","last_active":"2024-04-24T16:20:15.879433+0000","last_peered":"2024-04-24T16:20:15.879433+0000","last_clean":"2024-04-24T16:20:15.879433+0000","last_became_active":"2024-04-24T16:20:04.808494+0000","last_became_peered":"2024-04-24T16:20:04.808494+0000","last_unstale":"2024-04-24T16:20:15.879433+0000","last_undegraded":"2024-04-24T16:20:15.879433+0000","last_fullsized":"2024-04-24T16:20:15.879433+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T16:18:11.266384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T16:18:11.266384+0000","last_clean_scrub_stamp":"2024-04-24T16:18:11.266384+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:27:16.795622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.378},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46700000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.51100000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.44900000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.496},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.41799999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42999999999999999}]},{"osd":6,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.51800000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.5},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48599999999999999}]},{"osd":3,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.439},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.628}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.61799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.33800000000000002,"15min":0.313},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.36099999999999999,"15min":0.33100000000000002},"min":{"1min":0.26700000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:21:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38400000000000001,"15min":0.376},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.38},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.42299999999999999,"15min":0.41199999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.432,"15min":0.42399999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41899999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.441},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.753,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.48599999999999999}]},{"osd":5,"last update":"Wed Apr 24 16:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.72099999999999997,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.622,"15min":0.626},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81899999999999995,"5min":1.004,"15min":1.004},"last":0.51100000000000001}]},{"osd":6,"last update":"Wed Apr 24 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.52000000000000002}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.33500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.379},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35799999999999998}]},{"osd":2,"last update":"Wed Apr 24 16:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.378,"15min":0.36799999999999999},"min":{"1min":0.26400000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52200000000000002,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.38},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38100000000000001,"15min":0.379},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.47799999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.28999999999999998}]},{"osd":3,"last update":"Wed Apr 24 16:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.29999999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64000000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.316},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.432,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.33100000000000002}]},{"osd":4,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79700000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86299999999999999,"5min":0.876,"15min":0.876},"last":0.47699999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.55900000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.505}]},{"osd":6,"last update":"Wed Apr 24 16:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.438},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.48499999999999999}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:21:10 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.34100000000000003,"15min":0.33600000000000002},"min":{"1min":0.22600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.255},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.35999999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.27800000000000002}]},{"osd":1,"last update":"Wed Apr 24 16:21:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38,"15min":0.378},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55300000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.371},{"interface":"front","average":{"1min":0.378,"5min":0.38300000000000001,"15min":0.38},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52900000000000003,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.32000000000000001}]},{"osd":3,"last update":"Wed Apr 24 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53300000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.46600000000000003,"15min":0.47199999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.35699999999999998}]},{"osd":4,"last update":"Wed Apr 24 16:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72499999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.37,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.61299999999999999}]},{"osd":5,"last update":"Wed Apr 24 16:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.82899999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.441,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.67000000000000004}]},{"osd":6,"last update":"Wed Apr 24 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.73999999999999999}]},{"osd":7,"last update":"Wed Apr 24 16:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.72099999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41599999999999998,"15min":0.41999999999999998},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42899999999999999,"15min":0.437},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.377}]},{"osd":2,"last update":"Wed Apr 24 16:20:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.308},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.59199999999999997,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.33500000000000002}]},{"osd":4,"last update":"Wed Apr 24 16:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64000000000000001,"15min":0.65400000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88600000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.62,"15min":0.63},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.502}]},{"osd":5,"last update":"Wed Apr 24 16:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59899999999999998,"15min":0.60899999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.84799999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93200000000000005,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.52000000000000002}]},{"osd":6,"last update":"Wed Apr 24 16:20:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.44600000000000001}]},{"osd":7,"last update":"Wed Apr 24 16:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.53000000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69499999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.55900000000000005,"15min":0.55500000000000005},"min":{"1min":0.36799999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57799999999999996,"15min":0.57899999999999996},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.71199999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54500000000000004}]},{"osd":3,"last update":"Wed Apr 24 16:20:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.47699999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77900000000000003,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.42299999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.41399999999999998}]},{"osd":5,"last update":"Wed Apr 24 16:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.41699999999999998,"15min":0.40200000000000002},"min":{"1min":0.33800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.32800000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.35499999999999998}]},{"osd":6,"last update":"Wed Apr 24 16:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.312}]},{"osd":7,"last update":"Wed Apr 24 16:21:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.33300000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":239865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 16:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.41299999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Apr 24 16:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.752,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.505},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.371,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Apr 24 16:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68500000000000005,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60199999999999998}]},{"osd":3,"last update":"Wed Apr 24 16:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73799999999999999,"5min":0.872,"15min":0.872},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.374,"5min":0.371,"15min":0.371},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53100000000000003}]},{"osd":4,"last update":"Wed Apr 24 16:21:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.39100000000000001,"15min":0.375},"min":{"1min":0.33500000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.42599999999999999,"15min":0.41199999999999998},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Wed Apr 24 16:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.39600000000000002}]},{"osd":7,"last update":"Wed Apr 24 16:21:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T16:21:26.802 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-24T16:21:26.802 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T16:21:26.802 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-24T16:21:26.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-24T16:21:26.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:26 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:21:26.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:26 smithi156 ceph-mon[22679]: Deploying daemon grafana.a on smithi156 2024-04-24T16:21:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T16:21:27.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:27.608Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003545219s 2024-04-24T16:21:27.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:27 smithi070 ceph-mon[21827]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:27.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:27 smithi070 ceph-mon[21827]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T16:21:27.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:27 smithi070 ceph-mon[26293]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:27.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:27 smithi070 ceph-mon[26293]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T16:21:27.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:27 smithi156 ceph-mon[22679]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:27.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:27 smithi156 ceph-mon[22679]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T16:21:28.763 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:21:28.763 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T16:21:29.372 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-24T16:21:29.373 INFO:tasks.cephadm:Setup complete, yielding 2024-04-24T16:21:29.373 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T16:21:29.388 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-04-24T16:21:29.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-24T16:21:29.412 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:21:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:29] "GET /metrics HTTP/1.1" 200 191494 "" "Prometheus/2.33.4" 2024-04-24T16:21:29.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:29 smithi070 ceph-mon[21827]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:29.693 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:29 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3720336360' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T16:21:29.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:29 smithi070 ceph-mon[26293]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:29.694 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:29 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3720336360' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T16:21:29.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:29 smithi156 ceph-mon[22679]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:29.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:29 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3720336360' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T16:21:31.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:31 smithi156 ceph-mon[22679]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:32.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:31 smithi070 ceph-mon[26293]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:32.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:31 smithi070 ceph-mon[21827]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:32.655 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:21:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "id": "a7834811-14c2-4a05-bd68-379dec49f2c7", 2024-04-24T16:21:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "r", 2024-04-24T16:21:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "current_period": "b233e901-2482-487e-b6b9-d4ae630da625", 2024-04-24T16:21:32.655 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 1 2024-04-24T16:21:32.655 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:21:32.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:21:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:21:32.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:32 smithi156 ceph-mon[22679]: osdmap e61: 8 total, 8 up, 8 in 2024-04-24T16:21:32.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:32 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3683045885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-24T16:21:32.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:32 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-24T16:21:32.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:32 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-24T16:21:32.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:32 smithi156 ceph-mon[22679]: osdmap e62: 8 total, 8 up, 8 in 2024-04-24T16:21:33.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[26293]: osdmap e61: 8 total, 8 up, 8 in 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3683045885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[26293]: osdmap e62: 8 total, 8 up, 8 in 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[21827]: osdmap e61: 8 total, 8 up, 8 in 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3683045885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-24T16:21:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:32 smithi070 ceph-mon[21827]: osdmap e62: 8 total, 8 up, 8 in 2024-04-24T16:21:33.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-24T16:21:33.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:33 smithi070 ceph-mon[21827]: pgmap v42: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:33.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:33 smithi070 ceph-mon[21827]: osdmap e63: 8 total, 8 up, 8 in 2024-04-24T16:21:33.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:33 smithi070 ceph-mon[26293]: pgmap v42: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:33.784 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:33 smithi070 ceph-mon[26293]: osdmap e63: 8 total, 8 up, 8 in 2024-04-24T16:21:33.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:33 smithi156 ceph-mon[22679]: pgmap v42: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:33.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:33 smithi156 ceph-mon[22679]: osdmap e63: 8 total, 8 up, 8 in 2024-04-24T16:21:35.204 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "id": "c1c4c292-73dc-4f00-a65c-ac4b149bbe96", 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default", 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "api_name": "default", 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "is_master": "true", 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "endpoints": [], 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "hostnames": [], 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "hostnames_s3website": [], 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "master_zone": "", 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "zones": [], 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "placement_targets": [], 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "default_placement": "", 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "realm_id": "a7834811-14c2-4a05-bd68-379dec49f2c7", 2024-04-24T16:21:35.205 INFO:teuthology.orchestra.run.smithi070.stdout: "sync_policy": { 2024-04-24T16:21:35.206 INFO:teuthology.orchestra.run.smithi070.stdout: "groups": [] 2024-04-24T16:21:35.206 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:35.206 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:21:35.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-24T16:21:35.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:35 smithi156 ceph-mon[22679]: pgmap v45: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:35.989 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:35 smithi070 ceph-mon[21827]: pgmap v45: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:35.990 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:35 smithi070 ceph-mon[26293]: pgmap v45: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:37.427 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:21:37.427 INFO:teuthology.orchestra.run.smithi070.stdout: "id": "0240f09c-8177-4ea4-b77b-ced26b574d61", 2024-04-24T16:21:37.427 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "z", 2024-04-24T16:21:37.427 INFO:teuthology.orchestra.run.smithi070.stdout: "domain_root": "z.rgw.meta:root", 2024-04-24T16:21:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "control_pool": "z.rgw.control", 2024-04-24T16:21:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-24T16:21:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-24T16:21:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "log_pool": "z.rgw.log", 2024-04-24T16:21:37.428 INFO:teuthology.orchestra.run.smithi070.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "otp_pool": "z.rgw.otp", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "system_key": { 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "access_key": "", 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: "secret_key": "" 2024-04-24T16:21:37.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: "placement_pools": [ 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "default-placement", 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: "val": { 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_classes": { 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: "STANDARD": { 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-24T16:21:37.430 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:37.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:37.433 INFO:teuthology.orchestra.run.smithi070.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-24T16:21:37.434 INFO:teuthology.orchestra.run.smithi070.stdout: "index_type": 0 2024-04-24T16:21:37.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:37.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:37.434 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:21:37.434 INFO:teuthology.orchestra.run.smithi070.stdout: "realm_id": "a7834811-14c2-4a05-bd68-379dec49f2c7", 2024-04-24T16:21:37.434 INFO:teuthology.orchestra.run.smithi070.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-24T16:21:37.434 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:21:37.819 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:37 smithi156 ceph-mon[22679]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:37.964 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:37 smithi070 ceph-mon[26293]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:37.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:37 smithi070 ceph-mon[21827]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:21:37.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-24T16:21:38.073 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 podman[38827]: 2024-04-24 16:21:38.047984457 +0000 UTC m=+0.341849635 container start 5a7c9a93bc9edf2b857d4b45cf4d91befaeb2859b99f85e41caeaed5de0c0090 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, io.buildah.version=1.24.2, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, release=236.1648460182, version=8.5, description=Ceph Grafana Container, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, vcs-type=git, build-date=2022-03-28T10:36:18.413762) 2024-04-24T16:21:38.384 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-24T16:21:38.384 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-24T16:21:38.384 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-24T16:21:38.384 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-24T16:21:38.384 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="App mode production" logger=settings 2024-04-24T16:21:38.385 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-24T16:21:38.386 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-24T16:21:38.386 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-24T16:21:38.386 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-24T16:21:38.386 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-24T16:21:38.386 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-24T16:21:38.387 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-24T16:21:38.388 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-24T16:21:38.389 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-24T16:21:38.390 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-24T16:21:38.391 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-24T16:21:38.392 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-24T16:21:38.392 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-24T16:21:38.392 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-24T16:21:38.393 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-24T16:21:38.393 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-24T16:21:38.393 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-24T16:21:38.393 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-24T16:21:38.394 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-24T16:21:38.395 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 bash[38827]: 5a7c9a93bc9edf2b857d4b45cf4d91befaeb2859b99f85e41caeaed5de0c0090 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-24T16:21:38.396 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-24T16:21:38.397 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-24T16:21:38.397 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-24T16:21:38.397 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-24T16:21:38.397 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-24T16:21:38.397 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 systemd[1]: Started Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:21:38.397 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-24T16:21:38.397 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-24T16:21:38.398 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-24T16:21:38.399 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-24T16:21:38.400 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-24T16:21:38.401 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-24T16:21:38.402 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-24T16:21:38.403 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-24T16:21:38.404 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-24T16:21:38.405 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-24T16:21:38.406 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-24T16:21:38.407 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-24T16:21:38.408 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-24T16:21:38.408 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-24T16:21:38.408 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-24T16:21:38.408 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-24T16:21:38.408 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-24T16:21:38.408 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-24T16:21:38.408 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-24T16:21:38.409 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-24T16:21:38.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-24T16:21:38.411 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-24T16:21:38.412 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-24T16:21:38.413 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-24T16:21:38.414 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-24T16:21:38.415 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-24T16:21:38.416 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-24T16:21:38.417 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-24T16:21:38.418 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-24T16:21:38.419 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=172.529371ms 2024-04-24T16:21:38.420 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-24T16:21:38.420 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-24T16:21:38.420 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-24T16:21:38.420 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-24T16:21:38.420 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-24T16:21:38.420 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-24T16:21:38.420 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-24T16:21:38.421 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-24T16:21:38.421 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-24T16:21:38.421 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-24T16:21:38.421 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:21:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:21:38+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-24T16:21:39.328 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[21827]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 660 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-24T16:21:39.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:39.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:39.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:39.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:39.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[26293]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 660 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-24T16:21:39.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:39.329 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:39.330 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:39.330 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:39.484 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:39 smithi156 ceph-mon[22679]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 660 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-24T16:21:39.484 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:39 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:39.484 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:39 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:39.484 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:39 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:39.484 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:39 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:39.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:21:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:39] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-04-24T16:21:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:41 smithi156 ceph-mon[22679]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-24T16:21:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:41 smithi156 ceph-mon[22679]: osdmap e64: 8 total, 8 up, 8 in 2024-04-24T16:21:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:41 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-24T16:21:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:41 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:41.478 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[26293]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-24T16:21:41.479 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[26293]: osdmap e64: 8 total, 8 up, 8 in 2024-04-24T16:21:41.479 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-24T16:21:41.479 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:41.479 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[21827]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-24T16:21:41.479 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[21827]: osdmap e64: 8 total, 8 up, 8 in 2024-04-24T16:21:41.479 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-24T16:21:41.479 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:41 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: osdmap e65: 8 total, 8 up, 8 in 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: Reconfiguring daemon alertmanager.a on smithi070 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: osdmap e66: 8 total, 8 up, 8 in 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-24T16:21:42.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: osdmap e65: 8 total, 8 up, 8 in 2024-04-24T16:21:42.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-24T16:21:42.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: Reconfiguring daemon alertmanager.a on smithi070 2024-04-24T16:21:42.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: osdmap e66: 8 total, 8 up, 8 in 2024-04-24T16:21:42.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:42 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-24T16:21:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-24T16:21:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: osdmap e65: 8 total, 8 up, 8 in 2024-04-24T16:21:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-24T16:21:42.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: Reconfiguring daemon alertmanager.a on smithi070 2024-04-24T16:21:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: osdmap e66: 8 total, 8 up, 8 in 2024-04-24T16:21:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-24T16:21:43.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:21:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:21:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:43 smithi070 ceph-mon[26293]: pgmap v51: 65 pgs: 26 unknown, 39 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-24T16:21:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:43 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-24T16:21:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:43 smithi070 ceph-mon[26293]: osdmap e67: 8 total, 8 up, 8 in 2024-04-24T16:21:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:43 smithi070 ceph-mon[21827]: pgmap v51: 65 pgs: 26 unknown, 39 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-24T16:21:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:43 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-24T16:21:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:43 smithi070 ceph-mon[21827]: osdmap e67: 8 total, 8 up, 8 in 2024-04-24T16:21:44.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:43 smithi156 ceph-mon[22679]: pgmap v51: 65 pgs: 26 unknown, 39 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-24T16:21:44.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:43 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3282067832' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-24T16:21:44.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:43 smithi156 ceph-mon[22679]: osdmap e67: 8 total, 8 up, 8 in 2024-04-24T16:21:44.739 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:44 smithi070 systemd[1]: Stopping Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:21:44.739 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:44 smithi070 bash[46133]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-24T16:21:45.006 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[43933]: level=info ts=2024-04-24T16:21:44.741Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:21:45.006 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:44 smithi070 podman[46146]: 2024-04-24 16:21:44.762058493 +0000 UTC m=+0.234845215 container died 4148de6cdaf1d83c2e1f63d0ab2b882c85f4fa0b4c7a512275df12191d1cca3b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:45.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[21827]: pgmap v54: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-24T16:21:45.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[21827]: osdmap e68: 8 total, 8 up, 8 in 2024-04-24T16:21:45.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-24T16:21:45.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-24T16:21:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[26293]: pgmap v54: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-24T16:21:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[26293]: osdmap e68: 8 total, 8 up, 8 in 2024-04-24T16:21:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-24T16:21:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:45 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-24T16:21:45.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 podman[46146]: 2024-04-24 16:21:45.141503543 +0000 UTC m=+0.614290262 container cleanup 4148de6cdaf1d83c2e1f63d0ab2b882c85f4fa0b4c7a512275df12191d1cca3b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:45.292 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 bash[46146]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a 2024-04-24T16:21:45.292 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 bash[46172]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-24T16:21:45.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:45 smithi156 ceph-mon[22679]: pgmap v54: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-24T16:21:45.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:45 smithi156 ceph-mon[22679]: osdmap e68: 8 total, 8 up, 8 in 2024-04-24T16:21:45.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:45 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-24T16:21:45.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:45 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-24T16:21:45.699 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 podman[46160]: 2024-04-24 16:21:45.409364563 +0000 UTC m=+0.647426985 container remove 4148de6cdaf1d83c2e1f63d0ab2b882c85f4fa0b4c7a512275df12191d1cca3b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:45.699 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 podman[46160]: 2024-04-24 16:21:45.468251392 +0000 UTC m=+0.706313769 volume remove d197124a53f76eaa5486a764d39a15c64bef3b5be4b33ab82c38a72f26e5bf60 2024-04-24T16:21:45.699 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-24T16:21:45.699 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 systemd[1]: Stopped Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:21:45.699 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 systemd[1]: Starting Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:21:45.951 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 podman[46243]: 2024-04-24 16:21:45.702251671 +0000 UTC m=+0.111487005 volume create 8c4b1742d781d373289ff61936abee73491fb2a959c6ba8eb7bf3070e66d2a86 2024-04-24T16:21:45.951 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 podman[46243]: 2024-04-24 16:21:45.60864631 +0000 UTC m=+0.017881649 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-24T16:21:45.951 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:45 smithi070 podman[46243]: 2024-04-24 16:21:45.777786334 +0000 UTC m=+0.187021658 container create 771467ffa56d57a1df2fe73b218566992ccbf654831211317dcab3c0c8cae2b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:46.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: osdmap e69: 8 total, 8 up, 8 in 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: osdmap e70: 8 total, 8 up, 8 in 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-24T16:21:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: osdmap e69: 8 total, 8 up, 8 in 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: osdmap e70: 8 total, 8 up, 8 in 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-24T16:21:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-24T16:21:46.292 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 podman[46243]: 2024-04-24 16:21:46.039299295 +0000 UTC m=+0.448534618 container init 771467ffa56d57a1df2fe73b218566992ccbf654831211317dcab3c0c8cae2b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 podman[46243]: 2024-04-24 16:21:46.041983565 +0000 UTC m=+0.451218898 container start 771467ffa56d57a1df2fe73b218566992ccbf654831211317dcab3c0c8cae2b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.051Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.051Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.052Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.70 port=9094 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.053Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.079Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.079Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.081Z caller=main.go:518 msg=Listening address=:9093 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:46.082Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 bash[46243]: 771467ffa56d57a1df2fe73b218566992ccbf654831211317dcab3c0c8cae2b0 2024-04-24T16:21:46.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:46 smithi070 systemd[1]: Started Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:21:46.505 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-24T16:21:46.505 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: osdmap e69: 8 total, 8 up, 8 in 2024-04-24T16:21:46.505 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-24T16:21:46.505 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-24T16:21:46.505 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:46.505 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-24T16:21:46.506 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: osdmap e70: 8 total, 8 up, 8 in 2024-04-24T16:21:46.506 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3082938762' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-24T16:21:46.506 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:46 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-24T16:21:47.250 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:21:47.250 INFO:teuthology.orchestra.run.smithi070.stdout: "id": "0532d1a4-67b6-4775-b16a-2333141ed432", 2024-04-24T16:21:47.250 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 1, 2024-04-24T16:21:47.250 INFO:teuthology.orchestra.run.smithi070.stdout: "predecessor_uuid": "b233e901-2482-487e-b6b9-d4ae630da625", 2024-04-24T16:21:47.250 INFO:teuthology.orchestra.run.smithi070.stdout: "sync_status": [], 2024-04-24T16:21:47.250 INFO:teuthology.orchestra.run.smithi070.stdout: "period_map": { 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "id": "0532d1a4-67b6-4775-b16a-2333141ed432", 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "zonegroups": [ 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "id": "c1c4c292-73dc-4f00-a65c-ac4b149bbe96", 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default", 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "api_name": "default", 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "is_master": "true", 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "endpoints": [], 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "hostnames": [], 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "hostnames_s3website": [], 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "master_zone": "0240f09c-8177-4ea4-b77b-ced26b574d61", 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "zones": [ 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-24T16:21:47.251 INFO:teuthology.orchestra.run.smithi070.stdout: "id": "0240f09c-8177-4ea4-b77b-ced26b574d61", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "z", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "endpoints": [], 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "log_meta": "false", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "log_data": "false", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "bucket_index_max_shards": 11, 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "read_only": "false", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "tier_type": "", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "sync_from_all": "true", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "sync_from": [], 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect_zone": "" 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "placement_targets": [ 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default-placement", 2024-04-24T16:21:47.252 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_classes": [ 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: "STANDARD" 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: "default_placement": "default-placement", 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: "realm_id": "a7834811-14c2-4a05-bd68-379dec49f2c7", 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: "sync_policy": { 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: "groups": [] 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: "short_zone_ids": [ 2024-04-24T16:21:47.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "0240f09c-8177-4ea4-b77b-ced26b574d61", 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 2127864855 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: "master_zonegroup": "c1c4c292-73dc-4f00-a65c-ac4b149bbe96", 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: "master_zone": "0240f09c-8177-4ea4-b77b-ced26b574d61", 2024-04-24T16:21:47.254 INFO:teuthology.orchestra.run.smithi070.stdout: "period_config": { 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "bucket_quota": { 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "user_quota": { 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false, 2024-04-24T16:21:47.258 INFO:teuthology.orchestra.run.smithi070.stdout: "check_on_raw": false, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size": -1, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_size_kb": 0, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_objects": -1 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "user_ratelimit": { 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_read_ops": 0, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_write_ops": 0, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_read_bytes": 0, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_write_bytes": 0, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "bucket_ratelimit": { 2024-04-24T16:21:47.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max_read_ops": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max_write_ops": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max_read_bytes": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max_write_bytes": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "anonymous_ratelimit": { 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max_read_ops": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max_write_ops": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max_read_bytes": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max_write_bytes": 0, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": false 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:21:47.260 INFO:teuthology.orchestra.run.smithi070.stdout: "realm_id": "a7834811-14c2-4a05-bd68-379dec49f2c7", 2024-04-24T16:21:47.261 INFO:teuthology.orchestra.run.smithi070.stdout: "realm_name": "r", 2024-04-24T16:21:47.261 INFO:teuthology.orchestra.run.smithi070.stdout: "realm_epoch": 2 2024-04-24T16:21:47.261 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:21:47.472 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[21827]: pgmap v57: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[21827]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[21827]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[21827]: osdmap e71: 8 total, 8 up, 8 in 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[26293]: pgmap v57: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[26293]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[26293]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-24T16:21:47.473 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:47 smithi070 ceph-mon[26293]: osdmap e71: 8 total, 8 up, 8 in 2024-04-24T16:21:47.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:47 smithi156 ceph-mon[22679]: pgmap v57: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-24T16:21:47.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:47 smithi156 ceph-mon[22679]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:21:47.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:47 smithi156 ceph-mon[22679]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:21:47.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:47 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-24T16:21:47.510 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:47 smithi156 ceph-mon[22679]: osdmap e71: 8 total, 8 up, 8 in 2024-04-24T16:21:47.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-24T16:21:48.138 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:48.054Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000504218s 2024-04-24T16:21:49.552 INFO:teuthology.orchestra.run.smithi070.stdout:Scheduled rgw.foo update... 2024-04-24T16:21:49.609 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:21:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:49] "GET /metrics HTTP/1.1" 200 196351 "" "Prometheus/2.33.4" 2024-04-24T16:21:49.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:49 smithi156 ceph-mon[22679]: pgmap v60: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-24T16:21:49.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:49 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:49.979 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:49 smithi070 ceph-mon[26293]: pgmap v60: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-24T16:21:49.979 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:49 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:49.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:49 smithi070 ceph-mon[21827]: pgmap v60: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-24T16:21:49.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:49 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:50.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph osd pool create foo' 2024-04-24T16:21:50.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:50 smithi156 ceph-mon[22679]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:21:50.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:50 smithi156 ceph-mon[22679]: Saving service rgw.foo spec with placement count:2 2024-04-24T16:21:51.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:50 smithi070 ceph-mon[21827]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:21:51.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:50 smithi070 ceph-mon[21827]: Saving service rgw.foo spec with placement count:2 2024-04-24T16:21:51.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:50 smithi070 ceph-mon[26293]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:21:51.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:50 smithi070 ceph-mon[26293]: Saving service rgw.foo spec with placement count:2 2024-04-24T16:21:51.655 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:51 smithi070 ceph-mon[26293]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 181 B/s rd, 363 B/s wr, 1 op/s 2024-04-24T16:21:51.655 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:51 smithi070 ceph-mon[21827]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 181 B/s rd, 363 B/s wr, 1 op/s 2024-04-24T16:21:51.802 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:51 smithi156 ceph-mon[22679]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 181 B/s rd, 363 B/s wr, 1 op/s 2024-04-24T16:21:52.632 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'foo' created 2024-04-24T16:21:52.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:21:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:21:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:21:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:52 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2578974678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-24T16:21:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:52 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-24T16:21:52.926 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:52 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2578974678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-24T16:21:52.926 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:52 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-24T16:21:52.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:52 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2578974678' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-24T16:21:52.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:52 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-24T16:21:53.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'rbd pool init foo' 2024-04-24T16:21:53.523 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 bash[39455]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:21:53.840 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-mon[22679]: pgmap v62: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 308 B/s rd, 462 B/s wr, 1 op/s 2024-04-24T16:21:53.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-24T16:21:53.841 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-mon[22679]: osdmap e72: 8 total, 8 up, 8 in 2024-04-24T16:21:53.841 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:21:53.841 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-24T16:21:53.841 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-24T16:21:53.841 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-24T16:21:53.841 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-24T16:21:53.841 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-24T16:21:53.842 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-24T16:21:53.842 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-24T16:21:53.842 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-24T16:21:53.842 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.526Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-24T16:21:53.842 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.527Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-24T16:21:53.842 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[37807]: ts=2024-04-24T16:21:53.527Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-24T16:21:53.842 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 podman[39468]: 2024-04-24 16:21:53.536371967 +0000 UTC m=+0.152689312 container died 343ff88840beb55a5beb3c2b9d7354f6a51d8690def0ebc7b244033f7e4126f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:53.972 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:53 smithi070 ceph-mon[21827]: pgmap v62: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 308 B/s rd, 462 B/s wr, 1 op/s 2024-04-24T16:21:53.973 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:53 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-24T16:21:53.973 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:53 smithi070 ceph-mon[21827]: osdmap e72: 8 total, 8 up, 8 in 2024-04-24T16:21:53.973 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:53 smithi070 ceph-mon[26293]: pgmap v62: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 308 B/s rd, 462 B/s wr, 1 op/s 2024-04-24T16:21:53.973 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:53 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-24T16:21:53.973 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:53 smithi070 ceph-mon[26293]: osdmap e72: 8 total, 8 up, 8 in 2024-04-24T16:21:54.107 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 podman[39468]: 2024-04-24 16:21:53.942707108 +0000 UTC m=+0.559024447 container cleanup 343ff88840beb55a5beb3c2b9d7354f6a51d8690def0ebc7b244033f7e4126f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:54.107 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 bash[39468]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a 2024-04-24T16:21:54.107 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:53 smithi156 bash[39495]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:21:54.408 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 podman[39482]: 2024-04-24 16:21:54.110285896 +0000 UTC m=+0.575284140 container remove 343ff88840beb55a5beb3c2b9d7354f6a51d8690def0ebc7b244033f7e4126f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:54.408 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-24T16:21:54.408 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 systemd[1]: Stopped Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:21:54.408 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 systemd[1]: Starting Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:21:54.408 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 podman[39565]: 2024-04-24 16:21:54.239805251 +0000 UTC m=+0.018619541 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-24T16:21:54.408 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 podman[39565]: 2024-04-24 16:21:54.336072801 +0000 UTC m=+0.114887083 container create 14c1603136ac17e876390d9937f667eadb2f2e505af9ceee1151fba5deaa5ba2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:54.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-mon[22679]: osdmap e73: 8 total, 8 up, 8 in 2024-04-24T16:21:54.727 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 podman[39565]: 2024-04-24 16:21:54.5549679 +0000 UTC m=+0.333782185 container init 14c1603136ac17e876390d9937f667eadb2f2e505af9ceee1151fba5deaa5ba2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:54.727 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 podman[39565]: 2024-04-24 16:21:54.557611086 +0000 UTC m=+0.336425372 container start 14c1603136ac17e876390d9937f667eadb2f2e505af9ceee1151fba5deaa5ba2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:21:54.727 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.581Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-24T16:21:54.727 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.581Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.581Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.581Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi156 (none))" 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.581Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.581Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.582Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.583Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.584Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.585Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.585Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.552µs 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.585Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-24T16:21:54.728 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 bash[39565]: 14c1603136ac17e876390d9937f667eadb2f2e505af9ceee1151fba5deaa5ba2 2024-04-24T16:21:54.926 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:54 smithi070 ceph-mon[26293]: osdmap e73: 8 total, 8 up, 8 in 2024-04-24T16:21:54.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:54 smithi070 ceph-mon[21827]: osdmap e73: 8 total, 8 up, 8 in 2024-04-24T16:21:55.158 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 systemd[1]: Started Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:21:55.158 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.799Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-24T16:21:55.158 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.799Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-24T16:21:55.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.799Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.644µs wal_replay_duration=213.764263ms total_replay_duration=213.805997ms 2024-04-24T16:21:55.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.801Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-24T16:21:55.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.801Z caller=main.go:947 level=info msg="TSDB started" 2024-04-24T16:21:55.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.801Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-24T16:21:55.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.818Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.445516ms db_storage=1.127µs remote_storage=1.29µs web_handler=381ns query_engine=709ns scrape=382.226µs scrape_sd=25.051µs notify=30.112µs notify_sd=13.723µs rules=15.740835ms 2024-04-24T16:21:55.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:21:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:21:54.818Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: pgmap v65: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: osdmap e74: 8 total, 8 up, 8 in 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.70:9093"}]: dispatch 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.70:9093"}]: dispatch 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:21:55.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.156:9095"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.156:9095"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.156:3000"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3492347616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-24T16:21:55.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: pgmap v65: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: osdmap e74: 8 total, 8 up, 8 in 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.70:9093"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.70:9093"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.156:9095"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.156:9095"}]: dispatch 2024-04-24T16:21:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.156:3000"}]: dispatch 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3492347616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-24T16:21:55.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:55 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: pgmap v65: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: osdmap e74: 8 total, 8 up, 8 in 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.70:9093"}]: dispatch 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.70:9093"}]: dispatch 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:21:55.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.156:9095"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.156:9095"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.156:3000"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3492347616' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-24T16:21:55.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:55 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-24T16:21:56.394 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:21:56.056Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003315863s 2024-04-24T16:21:56.631 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:56 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:21:56.631 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:56 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.156:3000"}]: dispatch 2024-04-24T16:21:56.631 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:56 smithi156 ceph-mon[22679]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-24T16:21:56.631 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:56 smithi156 ceph-mon[22679]: osdmap e75: 8 total, 8 up, 8 in 2024-04-24T16:21:56.631 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:56 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:56.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:21:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.156:3000"}]: dispatch 2024-04-24T16:21:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[21827]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-24T16:21:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[21827]: osdmap e75: 8 total, 8 up, 8 in 2024-04-24T16:21:56.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:56.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:21:56.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.156:3000"}]: dispatch 2024-04-24T16:21:56.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[26293]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-24T16:21:56.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[26293]: osdmap e75: 8 total, 8 up, 8 in 2024-04-24T16:21:56.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:56 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:57.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:57 smithi156 ceph-mon[22679]: pgmap v68: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-24T16:21:57.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:57 smithi156 ceph-mon[22679]: osdmap e76: 8 total, 8 up, 8 in 2024-04-24T16:21:57.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:57 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:58.014 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:57 smithi070 ceph-mon[26293]: pgmap v68: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-24T16:21:58.015 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:57 smithi070 ceph-mon[26293]: osdmap e76: 8 total, 8 up, 8 in 2024-04-24T16:21:58.015 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:57 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:58.015 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:57 smithi070 ceph-mon[21827]: pgmap v68: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-24T16:21:58.015 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:57 smithi070 ceph-mon[21827]: osdmap e76: 8 total, 8 up, 8 in 2024-04-24T16:21:58.015 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:57 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:58.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-24T16:21:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: osdmap e77: 8 total, 8 up, 8 in 2024-04-24T16:21:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-24T16:21:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: Saving service rgw.foo spec with placement count:2 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[26293]: Deploying daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:21:59.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: osdmap e77: 8 total, 8 up, 8 in 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: Saving service rgw.foo spec with placement count:2 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:21:59.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:21:59.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-24T16:21:59.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:59.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:21:58 smithi070 ceph-mon[21827]: Deploying daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:21:59.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: osdmap e77: 8 total, 8 up, 8 in 2024-04-24T16:21:59.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-24T16:21:59.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: Saving service rgw.foo spec with placement count:2 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:21:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:21:58 smithi156 ceph-mon[22679]: Deploying daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:22:01.025 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:00 smithi070 ceph-mon[26293]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 197 B/s wr, 0 op/s 2024-04-24T16:22:01.025 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:00 smithi070 ceph-mon[21827]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 197 B/s wr, 0 op/s 2024-04-24T16:22:01.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:00 smithi156 ceph-mon[22679]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 197 B/s wr, 0 op/s 2024-04-24T16:22:01.698 INFO:teuthology.orchestra.run.smithi070.stdout:Scheduled iscsi.foo update... 2024-04-24T16:22:02.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'sleep 180' 2024-04-24T16:22:02.862 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:22:02.862 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: Saving service iscsi.foo spec with placement count:1 2024-04-24T16:22:02.862 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.862 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.862 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: Saving service iscsi.foo spec with placement count:1 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:22:02.863 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:22:02.864 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-24T16:22:02.864 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.864 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:02 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:02.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:22:02.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:22:02.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: Saving service iscsi.foo spec with placement count:1 2024-04-24T16:22:02.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:22:02.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:22:02.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-24T16:22:02.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:02.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:02 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:03.898 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:03 smithi156 ceph-mon[22679]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 339 B/s wr, 0 op/s 2024-04-24T16:22:03.899 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:03 smithi156 ceph-mon[22679]: Deploying daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:22:04.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:03 smithi070 ceph-mon[26293]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 339 B/s wr, 0 op/s 2024-04-24T16:22:04.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:03 smithi070 ceph-mon[26293]: Deploying daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:22:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:03 smithi070 ceph-mon[21827]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 339 B/s wr, 0 op/s 2024-04-24T16:22:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:03 smithi070 ceph-mon[21827]: Deploying daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:22:06.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[26293]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-24T16:22:06.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:06.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:06.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:06.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:06.027 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[21827]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-24T16:22:06.027 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:06.027 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:06.027 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:06.027 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:05 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:06.076 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:05 smithi156 ceph-mon[22679]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-24T16:22:06.076 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:05 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:06.076 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:05 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:06.076 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:05 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:06.076 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:05 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:06 smithi156 ceph-mon[22679]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 226 B/s wr, 0 op/s 2024-04-24T16:22:06.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:06 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:07.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:06 smithi070 ceph-mon[26293]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 226 B/s wr, 0 op/s 2024-04-24T16:22:07.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:06 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:07.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:06 smithi070 ceph-mon[21827]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 226 B/s wr, 0 op/s 2024-04-24T16:22:07.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:06 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: osdmap e78: 8 total, 8 up, 8 in 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:22:08.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: osdmap e78: 8 total, 8 up, 8 in 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-24T16:22:08.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:08 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: osdmap e78: 8 total, 8 up, 8 in 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-24T16:22:08.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:08 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:09.523 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:09] "GET /metrics HTTP/1.1" 200 197873 "" "Prometheus/2.33.4" 2024-04-24T16:22:09.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[26293]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.5 KiB/s wr, 212 op/s 2024-04-24T16:22:09.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[26293]: Checking dashboard <-> RGW credentials 2024-04-24T16:22:09.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[26293]: Checking pool "foo" exists for service iscsi.foo 2024-04-24T16:22:09.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[26293]: Deploying daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:22:09.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[21827]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.5 KiB/s wr, 212 op/s 2024-04-24T16:22:09.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[21827]: Checking dashboard <-> RGW credentials 2024-04-24T16:22:09.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[21827]: Checking pool "foo" exists for service iscsi.foo 2024-04-24T16:22:09.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:09 smithi070 ceph-mon[21827]: Deploying daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:22:09.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:09 smithi156 ceph-mon[22679]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.5 KiB/s wr, 212 op/s 2024-04-24T16:22:09.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:09 smithi156 ceph-mon[22679]: Checking dashboard <-> RGW credentials 2024-04-24T16:22:09.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:09 smithi156 ceph-mon[22679]: Checking pool "foo" exists for service iscsi.foo 2024-04-24T16:22:09.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:09 smithi156 ceph-mon[22679]: Deploying daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:22:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:11 smithi156 ceph-mon[22679]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.5 KiB/s wr, 212 op/s 2024-04-24T16:22:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:11 smithi156 ceph-mon[22679]: mgrmap e20: y(active, since 114s), standbys: x 2024-04-24T16:22:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:11 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:11 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:11 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:11 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:11.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:11 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2816508099' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[26293]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.5 KiB/s wr, 212 op/s 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[26293]: mgrmap e20: y(active, since 114s), standbys: x 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2816508099' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:22:11.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[21827]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.5 KiB/s wr, 212 op/s 2024-04-24T16:22:11.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[21827]: mgrmap e20: y(active, since 114s), standbys: x 2024-04-24T16:22:11.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:11.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:11.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:11.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:11.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:11 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2816508099' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:22:12.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:12 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2886805230' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2767731331"}]: dispatch 2024-04-24T16:22:12.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:12 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2886805230' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2767731331"}]: dispatch 2024-04-24T16:22:12.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:12 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2886805230' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2767731331"}]: dispatch 2024-04-24T16:22:13.000 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:22:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:22:13.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[21827]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.7 KiB/s wr, 217 op/s 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2886805230' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2767731331"}]': finished 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[21827]: osdmap e79: 8 total, 8 up, 8 in 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2273074679' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]: dispatch 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]: dispatch 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[26293]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.7 KiB/s wr, 217 op/s 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2886805230' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2767731331"}]': finished 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[26293]: osdmap e79: 8 total, 8 up, 8 in 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2273074679' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]: dispatch 2024-04-24T16:22:13.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:13 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]: dispatch 2024-04-24T16:22:13.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:13 smithi156 ceph-mon[22679]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.7 KiB/s wr, 217 op/s 2024-04-24T16:22:13.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:13 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2886805230' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2767731331"}]': finished 2024-04-24T16:22:13.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:13 smithi156 ceph-mon[22679]: osdmap e79: 8 total, 8 up, 8 in 2024-04-24T16:22:13.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:13 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2273074679' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]: dispatch 2024-04-24T16:22:13.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:13 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]: dispatch 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]': finished 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: osdmap e80: 8 total, 8 up, 8 in 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1804095343' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]: dispatch 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]: dispatch 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi070"}]: dispatch 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:14.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:14.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:14.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:14 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]': finished 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: osdmap e80: 8 total, 8 up, 8 in 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1804095343' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]: dispatch 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]: dispatch 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi070"}]: dispatch 2024-04-24T16:22:15.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4043624147"}]': finished 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: osdmap e80: 8 total, 8 up, 8 in 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1804095343' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]: dispatch 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]: dispatch 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi070"}]: dispatch 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:22:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:14 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 494 B/s wr, 8 op/s 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: Adding iSCSI gateway http://:@172.21.15.70:5000 to Dashboard 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi070"}]: dispatch 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: Checking dashboard <-> RGW credentials 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]': finished 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: osdmap e81: 8 total, 8 up, 8 in 2024-04-24T16:22:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:15 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4166588837' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2490199536"}]: dispatch 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 494 B/s wr, 8 op/s 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: Adding iSCSI gateway http://:@172.21.15.70:5000 to Dashboard 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi070"}]: dispatch 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: Checking dashboard <-> RGW credentials 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]': finished 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: osdmap e81: 8 total, 8 up, 8 in 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4166588837' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2490199536"}]: dispatch 2024-04-24T16:22:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 494 B/s wr, 8 op/s 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: Adding iSCSI gateway http://:@172.21.15.70:5000 to Dashboard 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi070"}]: dispatch 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: Checking dashboard <-> RGW credentials 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2548077742"}]': finished 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: osdmap e81: 8 total, 8 up, 8 in 2024-04-24T16:22:16.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:15 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4166588837' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2490199536"}]: dispatch 2024-04-24T16:22:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4166588837' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2490199536"}]': finished 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: osdmap e82: 8 total, 8 up, 8 in 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-24T16:22:16.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-24T16:22:16.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-24T16:22:16.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:22:16.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:22:16.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:22:16.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:22:16.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:16.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:16 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1641645600' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4129610450"}]: dispatch 2024-04-24T16:22:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4166588837' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2490199536"}]': finished 2024-04-24T16:22:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: osdmap e82: 8 total, 8 up, 8 in 2024-04-24T16:22:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-24T16:22:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-24T16:22:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-24T16:22:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-24T16:22:17.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1641645600' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4129610450"}]: dispatch 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4166588837' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2490199536"}]': finished 2024-04-24T16:22:17.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: osdmap e82: 8 total, 8 up, 8 in 2024-04-24T16:22:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-24T16:22:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-24T16:22:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-24T16:22:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-24T16:22:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-24T16:22:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:22:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:22:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:22:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:22:17.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:22:17.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:16 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1641645600' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4129610450"}]: dispatch 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1641645600' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4129610450"}]': finished 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: osdmap e83: 8 total, 8 up, 8 in 2024-04-24T16:22:17.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:17 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1688394271' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1502360754"}]: dispatch 2024-04-24T16:22:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:22:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-04-24T16:22:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-24T16:22:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-04-24T16:22:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-24T16:22:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1641645600' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4129610450"}]': finished 2024-04-24T16:22:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: osdmap e83: 8 total, 8 up, 8 in 2024-04-24T16:22:18.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1688394271' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1502360754"}]: dispatch 2024-04-24T16:22:18.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:22:18.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-04-24T16:22:18.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-24T16:22:18.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-04-24T16:22:18.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-24T16:22:18.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1641645600' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4129610450"}]': finished 2024-04-24T16:22:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: osdmap e83: 8 total, 8 up, 8 in 2024-04-24T16:22:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:17 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1688394271' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1502360754"}]: dispatch 2024-04-24T16:22:18.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:18 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1688394271' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1502360754"}]': finished 2024-04-24T16:22:18.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:18 smithi156 ceph-mon[22679]: osdmap e84: 8 total, 8 up, 8 in 2024-04-24T16:22:18.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:18 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1739112253' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]: dispatch 2024-04-24T16:22:18.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:18 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]: dispatch 2024-04-24T16:22:19.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1688394271' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1502360754"}]': finished 2024-04-24T16:22:19.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[26293]: osdmap e84: 8 total, 8 up, 8 in 2024-04-24T16:22:19.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1739112253' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]: dispatch 2024-04-24T16:22:19.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]: dispatch 2024-04-24T16:22:19.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1688394271' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1502360754"}]': finished 2024-04-24T16:22:19.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[21827]: osdmap e84: 8 total, 8 up, 8 in 2024-04-24T16:22:19.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1739112253' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]: dispatch 2024-04-24T16:22:19.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:18 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]: dispatch 2024-04-24T16:22:19.642 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:19] "GET /metrics HTTP/1.1" 200 207999 "" "Prometheus/2.33.4" 2024-04-24T16:22:19.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:19 smithi156 ceph-mon[22679]: pgmap v88: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-24T16:22:19.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:19 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]': finished 2024-04-24T16:22:19.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:19 smithi156 ceph-mon[22679]: osdmap e85: 8 total, 8 up, 8 in 2024-04-24T16:22:19.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:19 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/739428358' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]: dispatch 2024-04-24T16:22:19.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:19 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]: dispatch 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[21827]: pgmap v88: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]': finished 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[21827]: osdmap e85: 8 total, 8 up, 8 in 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/739428358' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]: dispatch 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]: dispatch 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[26293]: pgmap v88: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1351576920"}]': finished 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[26293]: osdmap e85: 8 total, 8 up, 8 in 2024-04-24T16:22:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/739428358' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]: dispatch 2024-04-24T16:22:20.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:19 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]: dispatch 2024-04-24T16:22:20.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:20 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]': finished 2024-04-24T16:22:20.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:20 smithi156 ceph-mon[22679]: osdmap e86: 8 total, 8 up, 8 in 2024-04-24T16:22:20.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:20 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2813252136' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/3814823340"}]: dispatch 2024-04-24T16:22:21.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:20 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]': finished 2024-04-24T16:22:21.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:20 smithi070 ceph-mon[21827]: osdmap e86: 8 total, 8 up, 8 in 2024-04-24T16:22:21.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:20 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2813252136' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/3814823340"}]: dispatch 2024-04-24T16:22:21.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:20 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2897538162"}]': finished 2024-04-24T16:22:21.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:20 smithi070 ceph-mon[26293]: osdmap e86: 8 total, 8 up, 8 in 2024-04-24T16:22:21.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:20 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2813252136' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/3814823340"}]: dispatch 2024-04-24T16:22:21.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:21 smithi156 ceph-mon[22679]: pgmap v91: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-24T16:22:21.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:21 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2813252136' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/3814823340"}]': finished 2024-04-24T16:22:21.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:21 smithi156 ceph-mon[22679]: osdmap e87: 8 total, 8 up, 8 in 2024-04-24T16:22:21.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:21 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/434293731' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]: dispatch 2024-04-24T16:22:21.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:21 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]: dispatch 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[21827]: pgmap v91: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2813252136' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/3814823340"}]': finished 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[21827]: osdmap e87: 8 total, 8 up, 8 in 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/434293731' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]: dispatch 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]: dispatch 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[26293]: pgmap v91: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2813252136' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/3814823340"}]': finished 2024-04-24T16:22:22.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[26293]: osdmap e87: 8 total, 8 up, 8 in 2024-04-24T16:22:22.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/434293731' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]: dispatch 2024-04-24T16:22:22.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:21 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]: dispatch 2024-04-24T16:22:23.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:23.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]': finished 2024-04-24T16:22:23.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[26293]: osdmap e88: 8 total, 8 up, 8 in 2024-04-24T16:22:23.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3576784762' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2548077742"}]: dispatch 2024-04-24T16:22:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]': finished 2024-04-24T16:22:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[21827]: osdmap e88: 8 total, 8 up, 8 in 2024-04-24T16:22:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:22 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3576784762' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2548077742"}]: dispatch 2024-04-24T16:22:23.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:22:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:22:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:22 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:22 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/752255822"}]': finished 2024-04-24T16:22:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:22 smithi156 ceph-mon[22679]: osdmap e88: 8 total, 8 up, 8 in 2024-04-24T16:22:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:22 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3576784762' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2548077742"}]: dispatch 2024-04-24T16:22:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:23 smithi156 ceph-mon[22679]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 89 B/s, 0 objects/s recovering 2024-04-24T16:22:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:23 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3576784762' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2548077742"}]': finished 2024-04-24T16:22:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:23 smithi156 ceph-mon[22679]: osdmap e89: 8 total, 8 up, 8 in 2024-04-24T16:22:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:23 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1925062670' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/3814823340"}]: dispatch 2024-04-24T16:22:24.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[26293]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 89 B/s, 0 objects/s recovering 2024-04-24T16:22:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3576784762' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2548077742"}]': finished 2024-04-24T16:22:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[26293]: osdmap e89: 8 total, 8 up, 8 in 2024-04-24T16:22:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1925062670' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/3814823340"}]: dispatch 2024-04-24T16:22:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[21827]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 89 B/s, 0 objects/s recovering 2024-04-24T16:22:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3576784762' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2548077742"}]': finished 2024-04-24T16:22:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[21827]: osdmap e89: 8 total, 8 up, 8 in 2024-04-24T16:22:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:23 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1925062670' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/3814823340"}]: dispatch 2024-04-24T16:22:25.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:24 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1925062670' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/3814823340"}]': finished 2024-04-24T16:22:25.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:24 smithi070 ceph-mon[26293]: osdmap e90: 8 total, 8 up, 8 in 2024-04-24T16:22:25.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:24 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3970466854' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2944003385"}]: dispatch 2024-04-24T16:22:25.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:24 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1925062670' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/3814823340"}]': finished 2024-04-24T16:22:25.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:24 smithi070 ceph-mon[21827]: osdmap e90: 8 total, 8 up, 8 in 2024-04-24T16:22:25.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:24 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3970466854' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2944003385"}]: dispatch 2024-04-24T16:22:25.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:24 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1925062670' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/3814823340"}]': finished 2024-04-24T16:22:25.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:24 smithi156 ceph-mon[22679]: osdmap e90: 8 total, 8 up, 8 in 2024-04-24T16:22:25.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:24 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3970466854' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2944003385"}]: dispatch 2024-04-24T16:22:26.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[26293]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 89 B/s, 0 objects/s recovering 2024-04-24T16:22:26.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3970466854' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2944003385"}]': finished 2024-04-24T16:22:26.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[26293]: osdmap e91: 8 total, 8 up, 8 in 2024-04-24T16:22:26.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2942329050' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2944003385"}]: dispatch 2024-04-24T16:22:26.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[21827]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 89 B/s, 0 objects/s recovering 2024-04-24T16:22:26.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3970466854' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2944003385"}]': finished 2024-04-24T16:22:26.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[21827]: osdmap e91: 8 total, 8 up, 8 in 2024-04-24T16:22:26.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:25 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2942329050' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2944003385"}]: dispatch 2024-04-24T16:22:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:25 smithi156 ceph-mon[22679]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 89 B/s, 0 objects/s recovering 2024-04-24T16:22:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:25 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3970466854' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2944003385"}]': finished 2024-04-24T16:22:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:25 smithi156 ceph-mon[22679]: osdmap e91: 8 total, 8 up, 8 in 2024-04-24T16:22:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:25 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2942329050' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2944003385"}]: dispatch 2024-04-24T16:22:27.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:26 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2942329050' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2944003385"}]': finished 2024-04-24T16:22:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:26 smithi070 ceph-mon[26293]: osdmap e92: 8 total, 8 up, 8 in 2024-04-24T16:22:27.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:26 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/407508618' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/312180501"}]: dispatch 2024-04-24T16:22:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:26 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2942329050' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2944003385"}]': finished 2024-04-24T16:22:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:26 smithi070 ceph-mon[21827]: osdmap e92: 8 total, 8 up, 8 in 2024-04-24T16:22:27.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:26 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/407508618' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/312180501"}]: dispatch 2024-04-24T16:22:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:26 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2942329050' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2944003385"}]': finished 2024-04-24T16:22:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:26 smithi156 ceph-mon[22679]: osdmap e92: 8 total, 8 up, 8 in 2024-04-24T16:22:27.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:26 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/407508618' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/312180501"}]: dispatch 2024-04-24T16:22:28.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[26293]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/407508618' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/312180501"}]': finished 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[26293]: osdmap e93: 8 total, 8 up, 8 in 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1231679865' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]: dispatch 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]: dispatch 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[21827]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/407508618' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/312180501"}]': finished 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[21827]: osdmap e93: 8 total, 8 up, 8 in 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1231679865' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]: dispatch 2024-04-24T16:22:28.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:27 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]: dispatch 2024-04-24T16:22:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:27 smithi156 ceph-mon[22679]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:22:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:27 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/407508618' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/312180501"}]': finished 2024-04-24T16:22:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:27 smithi156 ceph-mon[22679]: osdmap e93: 8 total, 8 up, 8 in 2024-04-24T16:22:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:27 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1231679865' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]: dispatch 2024-04-24T16:22:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:27 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]: dispatch 2024-04-24T16:22:29.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:28 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]': finished 2024-04-24T16:22:29.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:28 smithi070 ceph-mon[21827]: osdmap e94: 8 total, 8 up, 8 in 2024-04-24T16:22:29.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:28 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]': finished 2024-04-24T16:22:29.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:28 smithi070 ceph-mon[26293]: osdmap e94: 8 total, 8 up, 8 in 2024-04-24T16:22:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:28 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2369226179"}]': finished 2024-04-24T16:22:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:28 smithi156 ceph-mon[22679]: osdmap e94: 8 total, 8 up, 8 in 2024-04-24T16:22:29.692 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:22:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:29] "GET /metrics HTTP/1.1" 200 207999 "" "Prometheus/2.33.4" 2024-04-24T16:22:30.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:29 smithi070 ceph-mon[26293]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:30.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:29 smithi070 ceph-mon[21827]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:29 smithi156 ceph-mon[22679]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:32.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:31 smithi070 ceph-mon[26293]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-04-24T16:22:32.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:31 smithi070 ceph-mon[21827]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-04-24T16:22:32.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:31 smithi156 ceph-mon[22679]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 0 op/s 2024-04-24T16:22:33.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:32 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:32 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:33.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:32 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:22:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:22:34.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:33 smithi070 ceph-mon[21827]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:22:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:33 smithi070 ceph-mon[26293]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:22:34.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:33 smithi156 ceph-mon[22679]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:22:36.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:35 smithi070 ceph-mon[21827]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:36.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:35 smithi070 ceph-mon[26293]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:36.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:35 smithi156 ceph-mon[22679]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:38.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:37 smithi070 ceph-mon[26293]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:22:38.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:37 smithi070 ceph-mon[21827]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:22:38.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:37 smithi156 ceph-mon[22679]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:22:39.709 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:22:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:39] "GET /metrics HTTP/1.1" 200 208025 "" "Prometheus/2.33.4" 2024-04-24T16:22:40.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:39 smithi070 ceph-mon[21827]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s 2024-04-24T16:22:40.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:39 smithi070 ceph-mon[26293]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s 2024-04-24T16:22:40.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:39 smithi156 ceph-mon[22679]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s 2024-04-24T16:22:42.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:41 smithi070 ceph-mon[21827]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:42.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:41 smithi070 ceph-mon[26293]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:42.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:41 smithi156 ceph-mon[22679]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:43.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:42 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:43.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:42 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:43.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:42 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:43.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:22:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:22:44.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:43 smithi070 ceph-mon[26293]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:43 smithi070 ceph-mon[21827]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:44.071 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:43 smithi156 ceph-mon[22679]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:46.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:45 smithi070 ceph-mon[26293]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:45 smithi070 ceph-mon[21827]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:46.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:45 smithi156 ceph-mon[22679]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:48.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:47 smithi070 ceph-mon[26293]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:48.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:47 smithi070 ceph-mon[21827]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:47 smithi156 ceph-mon[22679]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:49.725 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:22:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:49] "GET /metrics HTTP/1.1" 200 207974 "" "Prometheus/2.33.4" 2024-04-24T16:22:50.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:49 smithi070 ceph-mon[21827]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:50.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:49 smithi070 ceph-mon[26293]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:49 smithi156 ceph-mon[22679]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:52.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:51 smithi070 ceph-mon[26293]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:51 smithi070 ceph-mon[21827]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:51 smithi156 ceph-mon[22679]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:53.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:52 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:52 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:53.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:52 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:22:53.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:22:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:22:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:53 smithi070 ceph-mon[21827]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:54.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:53 smithi070 ceph-mon[26293]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:54.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:53 smithi156 ceph-mon[22679]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:22:56.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:55 smithi070 ceph-mon[26293]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:55 smithi070 ceph-mon[21827]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:56.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:55 smithi156 ceph-mon[22679]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:58.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:57 smithi070 ceph-mon[26293]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:58.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:57 smithi070 ceph-mon[21827]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:58.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:57 smithi156 ceph-mon[22679]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:22:59.741 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:22:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:22:59] "GET /metrics HTTP/1.1" 200 207974 "" "Prometheus/2.33.4" 2024-04-24T16:23:00.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:22:59 smithi070 ceph-mon[21827]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:00.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:22:59 smithi070 ceph-mon[26293]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:00.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:22:59 smithi156 ceph-mon[22679]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:02.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:01 smithi070 ceph-mon[21827]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:02.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:01 smithi070 ceph-mon[26293]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:02.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:01 smithi156 ceph-mon[22679]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:03.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:02 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:03.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:02 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:03.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:02 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:03.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:23:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:23:04.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:03 smithi070 ceph-mon[26293]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:03 smithi070 ceph-mon[21827]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:04.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:03 smithi156 ceph-mon[22679]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:06.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:05 smithi070 ceph-mon[26293]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:06.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:05 smithi070 ceph-mon[21827]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:06.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:05 smithi156 ceph-mon[22679]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:08.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:07 smithi070 ceph-mon[26293]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:08.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:07 smithi070 ceph-mon[21827]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:08.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:07 smithi156 ceph-mon[22679]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:09.758 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:23:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:09] "GET /metrics HTTP/1.1" 200 207912 "" "Prometheus/2.33.4" 2024-04-24T16:23:10.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:09 smithi070 ceph-mon[26293]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:10.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:09 smithi070 ceph-mon[21827]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:10.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:09 smithi156 ceph-mon[22679]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:12.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:11 smithi070 ceph-mon[26293]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:12.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:11 smithi070 ceph-mon[21827]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:12.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:11 smithi156 ceph-mon[22679]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:13.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:12 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:13.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:12 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:13.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:12 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:13.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:23:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:23:14.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:13 smithi070 ceph-mon[26293]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:14.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:13 smithi070 ceph-mon[21827]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:14.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:13 smithi156 ceph-mon[22679]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:14.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:23:14.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:23:14.861 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:14 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:23:14.861 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:23:14.861 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:23:14.861 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:14 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:23:14.862 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:23:14.862 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:23:14.862 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:14 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:23:16.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:15 smithi070 ceph-mon[21827]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:16.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:15 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:23:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:15 smithi070 ceph-mon[26293]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:15 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:23:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:15 smithi156 ceph-mon[22679]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:15 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:23:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:23:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:23:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:23:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:23:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:23:18.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:17 smithi070 ceph-mon[26293]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:17 smithi070 ceph-mon[21827]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:18.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:17 smithi156 ceph-mon[22679]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:19.780 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:23:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:19] "GET /metrics HTTP/1.1" 200 207921 "" "Prometheus/2.33.4" 2024-04-24T16:23:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:19 smithi070 ceph-mon[21827]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:19 smithi070 ceph-mon[26293]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:20.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:19 smithi156 ceph-mon[22679]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:22.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:21 smithi070 ceph-mon[26293]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:21 smithi070 ceph-mon[21827]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:22.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:21 smithi156 ceph-mon[22679]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:23.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:22 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:22 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:23.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:22 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:23.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:23:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:23:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:23 smithi070 ceph-mon[26293]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:23 smithi070 ceph-mon[21827]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:23 smithi156 ceph-mon[22679]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:25 smithi156 ceph-mon[22679]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:26.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:25 smithi070 ceph-mon[26293]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:26.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:25 smithi070 ceph-mon[21827]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:28.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:27 smithi156 ceph-mon[22679]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:28.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:27 smithi070 ceph-mon[26293]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:28.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:27 smithi070 ceph-mon[21827]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:29.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:23:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:29] "GET /metrics HTTP/1.1" 200 207921 "" "Prometheus/2.33.4" 2024-04-24T16:23:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:29 smithi156 ceph-mon[22679]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:30.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:29 smithi070 ceph-mon[26293]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:29 smithi070 ceph-mon[21827]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:32.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:31 smithi156 ceph-mon[22679]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:32.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:31 smithi070 ceph-mon[26293]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:31 smithi070 ceph-mon[21827]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:33.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:32 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:33.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:32 smithi156 ceph-mon[22679]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:23:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:23:33.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:32 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:33.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:32 smithi070 ceph-mon[26293]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:33.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:32 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:33.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:32 smithi070 ceph-mon[21827]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:34 smithi156 ceph-mon[22679]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:35.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:34 smithi070 ceph-mon[26293]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:34 smithi070 ceph-mon[21827]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:37.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:36 smithi070 ceph-mon[21827]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:37.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:36 smithi070 ceph-mon[26293]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:37.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:36 smithi156 ceph-mon[22679]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:39.633 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:23:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:39] "GET /metrics HTTP/1.1" 200 207937 "" "Prometheus/2.33.4" 2024-04-24T16:23:39.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:39 smithi156 ceph-mon[22679]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:39 smithi070 ceph-mon[26293]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:39 smithi070 ceph-mon[21827]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:41.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:41 smithi156 ceph-mon[22679]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:42.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:41 smithi070 ceph-mon[26293]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:42.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:41 smithi070 ceph-mon[21827]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:42.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:23:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:23:42.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:42 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:43.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:42 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:43.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:42 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:43.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:43 smithi156 ceph-mon[22679]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:44.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:43 smithi070 ceph-mon[21827]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:43 smithi070 ceph-mon[26293]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:45.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:45 smithi156 ceph-mon[22679]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:46.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:45 smithi070 ceph-mon[26293]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:45 smithi070 ceph-mon[21827]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:23:47.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:47 smithi156 ceph-mon[22679]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:48.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:47 smithi070 ceph-mon[26293]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:48.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:47 smithi070 ceph-mon[21827]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:49.649 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:23:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:49] "GET /metrics HTTP/1.1" 200 207936 "" "Prometheus/2.33.4" 2024-04-24T16:23:49.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:49 smithi156 ceph-mon[22679]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:50.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:49 smithi070 ceph-mon[26293]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:50.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:49 smithi070 ceph-mon[21827]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:51.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:51 smithi156 ceph-mon[22679]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:52.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:51 smithi070 ceph-mon[21827]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:52.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:51 smithi070 ceph-mon[26293]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:52 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:52.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:23:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:23:53.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:52 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:52 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:23:54.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:53 smithi070 ceph-mon[26293]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:53 smithi070 ceph-mon[21827]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:54.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:53 smithi156 ceph-mon[22679]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:23:56.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:55 smithi070 ceph-mon[26293]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:55 smithi070 ceph-mon[21827]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:56.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:55 smithi156 ceph-mon[22679]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:58.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:57 smithi070 ceph-mon[26293]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:58.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:57 smithi070 ceph-mon[21827]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:58.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:57 smithi156 ceph-mon[22679]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:23:59.666 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:23:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:23:59] "GET /metrics HTTP/1.1" 200 207936 "" "Prometheus/2.33.4" 2024-04-24T16:24:00.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:23:59 smithi070 ceph-mon[21827]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:00.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:23:59 smithi070 ceph-mon[26293]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:00.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:23:59 smithi156 ceph-mon[22679]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:02.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:01 smithi070 ceph-mon[21827]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:02.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:01 smithi070 ceph-mon[26293]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:02.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:01 smithi156 ceph-mon[22679]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:03.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:02 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:03.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:02 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:02 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:03.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:24:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:24:04.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:03 smithi070 ceph-mon[26293]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:03 smithi070 ceph-mon[21827]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:04.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:03 smithi156 ceph-mon[22679]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:05 smithi070 ceph-mon[26293]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:06.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:05 smithi070 ceph-mon[21827]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:06.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:05 smithi156 ceph-mon[22679]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:08.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:07 smithi070 ceph-mon[26293]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:08.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:07 smithi070 ceph-mon[21827]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:08.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:07 smithi156 ceph-mon[22679]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:09.682 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:24:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:09] "GET /metrics HTTP/1.1" 200 207936 "" "Prometheus/2.33.4" 2024-04-24T16:24:10.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:09 smithi070 ceph-mon[26293]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:10.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:09 smithi070 ceph-mon[21827]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:10.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:09 smithi156 ceph-mon[22679]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:12.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:11 smithi070 ceph-mon[21827]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:12.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:11 smithi070 ceph-mon[26293]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:12.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:11 smithi156 ceph-mon[22679]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:13.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:12 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:13.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:12 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:13.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:12 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:13.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:24:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:24:14.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:13 smithi070 ceph-mon[21827]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:14.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:13 smithi070 ceph-mon[26293]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:14.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:13 smithi156 ceph-mon[22679]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:16.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[21827]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[26293]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:24:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:15 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:24:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:15 smithi156 ceph-mon[22679]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:15 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:24:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:15 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:24:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:15 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:24:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:15 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:24:17.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:24:17.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:24:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:24:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:24:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:24:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:24:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:24:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:24:17.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:24:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:24:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:24:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:24:18.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:17 smithi070 ceph-mon[21827]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:18.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:17 smithi070 ceph-mon[26293]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:17 smithi156 ceph-mon[22679]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:19.708 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:24:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:19] "GET /metrics HTTP/1.1" 200 207925 "" "Prometheus/2.33.4" 2024-04-24T16:24:20.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:19 smithi070 ceph-mon[26293]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:19 smithi070 ceph-mon[21827]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:20.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:19 smithi156 ceph-mon[22679]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:22.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:21 smithi070 ceph-mon[26293]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:21 smithi070 ceph-mon[21827]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:22.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:21 smithi156 ceph-mon[22679]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:23.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:22 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:23.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:22 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:22 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:23.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:24:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:24:24.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:23 smithi070 ceph-mon[26293]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:24.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:23 smithi070 ceph-mon[21827]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:24.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:23 smithi156 ceph-mon[22679]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:26.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:25 smithi070 ceph-mon[21827]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:26.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:25 smithi070 ceph-mon[26293]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:26.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:25 smithi156 ceph-mon[22679]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:28.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:27 smithi070 ceph-mon[21827]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:28.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:27 smithi070 ceph-mon[26293]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:28.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:27 smithi156 ceph-mon[22679]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:29.725 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:24:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:29] "GET /metrics HTTP/1.1" 200 207925 "" "Prometheus/2.33.4" 2024-04-24T16:24:30.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:29 smithi070 ceph-mon[21827]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:30.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:29 smithi070 ceph-mon[26293]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:29 smithi156 ceph-mon[22679]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:32.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:31 smithi070 ceph-mon[26293]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:32.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:31 smithi070 ceph-mon[21827]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:32.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:31 smithi156 ceph-mon[22679]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:33.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:32 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:33.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:32 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:33.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:32 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:24:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:24:34.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:33 smithi070 ceph-mon[26293]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:34.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:33 smithi070 ceph-mon[21827]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:34.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:33 smithi156 ceph-mon[22679]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:36.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:35 smithi070 ceph-mon[21827]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:36.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:35 smithi070 ceph-mon[26293]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:36.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:35 smithi156 ceph-mon[22679]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:38.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:37 smithi070 ceph-mon[26293]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:38.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:37 smithi070 ceph-mon[21827]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:37 smithi156 ceph-mon[22679]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:39.741 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:24:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:39] "GET /metrics HTTP/1.1" 200 207907 "" "Prometheus/2.33.4" 2024-04-24T16:24:40.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:39 smithi070 ceph-mon[21827]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:40.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:39 smithi070 ceph-mon[26293]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:40.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:39 smithi156 ceph-mon[22679]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:42.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:41 smithi070 ceph-mon[26293]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:42.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:41 smithi070 ceph-mon[21827]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:42.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:41 smithi156 ceph-mon[22679]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:43.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:42 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:43.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:42 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:43.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:42 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:43.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:24:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:24:44.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:43 smithi070 ceph-mon[26293]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:43 smithi070 ceph-mon[21827]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:44.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:43 smithi156 ceph-mon[22679]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:46.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:45 smithi070 ceph-mon[26293]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:46.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:45 smithi070 ceph-mon[21827]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:46.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:45 smithi156 ceph-mon[22679]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:48.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:47 smithi070 ceph-mon[26293]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:48.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:47 smithi070 ceph-mon[21827]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:47 smithi156 ceph-mon[22679]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:49.757 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:24:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:49] "GET /metrics HTTP/1.1" 200 207874 "" "Prometheus/2.33.4" 2024-04-24T16:24:50.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:49 smithi070 ceph-mon[21827]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:50.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:49 smithi070 ceph-mon[26293]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:49 smithi156 ceph-mon[22679]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:52.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:51 smithi070 ceph-mon[26293]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:51 smithi070 ceph-mon[21827]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:52.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:51 smithi156 ceph-mon[22679]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:53.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:52 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:52 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:53.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:52 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:24:53.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:24:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:24:54.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:53 smithi070 ceph-mon[26293]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:53 smithi070 ceph-mon[21827]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:54.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:53 smithi156 ceph-mon[22679]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:24:56.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:55 smithi070 ceph-mon[26293]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:55 smithi070 ceph-mon[21827]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:56.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:55 smithi156 ceph-mon[22679]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:58.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:57 smithi070 ceph-mon[26293]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:58.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:57 smithi070 ceph-mon[21827]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:58.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:57 smithi156 ceph-mon[22679]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:24:59.773 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:24:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:24:59] "GET /metrics HTTP/1.1" 200 207874 "" "Prometheus/2.33.4" 2024-04-24T16:25:00.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:24:59 smithi070 ceph-mon[26293]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:24:59 smithi070 ceph-mon[21827]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:00.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:24:59 smithi156 ceph-mon[22679]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:02.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:01 smithi070 ceph-mon[26293]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:02.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:01 smithi070 ceph-mon[21827]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:02.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:01 smithi156 ceph-mon[22679]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:03.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:02 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:03.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:02 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:02 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:03.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:25:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:25:04.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:03 smithi070 ceph-mon[26293]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:03 smithi070 ceph-mon[21827]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:04.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:03 smithi156 ceph-mon[22679]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:04.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-24T16:25:05.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:05 smithi070 ceph-mon[21827]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:05.940 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:05 smithi070 ceph-mon[26293]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:06.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:05 smithi156 ceph-mon[22679]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:07.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-24T16:25:07.924 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:07 smithi070 ceph-mon[26293]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:07.925 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:07 smithi070 ceph-mon[21827]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:08.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:07 smithi156 ceph-mon[22679]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:09.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:25:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:09] "GET /metrics HTTP/1.1" 200 207878 "" "Prometheus/2.33.4" 2024-04-24T16:25:09.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-24T16:25:10.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:09 smithi070 ceph-mon[26293]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:10.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:09 smithi070 ceph-mon[21827]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:10.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:09 smithi156 ceph-mon[22679]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:12.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:11 smithi156 ceph-mon[22679]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:12.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:11 smithi070 ceph-mon[21827]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:12.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:11 smithi070 ceph-mon[26293]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:12.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:25:13.056 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:12 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:13.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:12 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:13.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:12 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:13.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:25:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:25:13.939 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:25:13.939 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 2m ago 3m 16.7M - ba2b418f427c 771467ffa56d 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (3m) 2m ago 3m 49.5M - 8.3.5 dad864ee21e9 5a7c9a93bc9e 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (3m) 2m ago 3m 62.5M - 3.5 e1d6a67b021e e257fa811608 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443 running (8m) 2m ago 8m 410M - 17.2.0 e1d6a67b021e df793845cb73 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:9283 running (9m) 2m ago 9m 454M - 17.2.0 e1d6a67b021e cbef4307b27d 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (9m) 2m ago 9m 45.0M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (8m) 2m ago 8m 41.2M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (8m) 2m ago 8m 39.4M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (4m) 2m ago 4m 15.0M - 1dbe0e931976 57518b11b60b 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (4m) 2m ago 4m 16.5M - 1dbe0e931976 a60500a59625 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (7m) 2m ago 7m 49.6M 4016M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (7m) 2m ago 7m 51.0M 4016M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (7m) 2m ago 7m 44.3M 4016M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (6m) 2m ago 6m 44.2M 4016M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (6m) 2m ago 6m 45.3M 4272M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:25:13.940 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (6m) 2m ago 6m 44.7M 4272M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:25:13.941 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (5m) 2m ago 5m 42.1M 4272M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:25:13.941 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (5m) 2m ago 5m 46.5M 4272M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:25:13.941 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (3m) 2m ago 4m 41.5M - 514e6a882f6e 14c1603136ac 2024-04-24T16:25:13.941 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (3m) 2m ago 3m 69.1M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:25:13.941 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (3m) 2m ago 3m 68.8M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:25:13.941 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:13 smithi070 ceph-mon[21827]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:13.941 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:13 smithi070 ceph-mon[26293]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:14.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:13 smithi156 ceph-mon[22679]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:14.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions' 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[26293]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[26293]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[21827]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[21827]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:15.975 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:25:15.976 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:25:15.976 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:15 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:25:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:15 smithi156 ceph-mon[22679]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:15 smithi156 ceph-mon[22679]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:15 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:25:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:15 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:25:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:15 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:25:16.410 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:25:16.411 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:25:16.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:25:16.411 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:25:16.411 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-04-24T16:25:16.411 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:25:16.411 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:25:17.001 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph -s' 2024-04-24T16:25:17.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:25:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:25:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:25:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:25:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:16 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/953003859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:25:17.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:25:17.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:25:17.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:25:17.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/953003859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:16 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/953003859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:25:17.943 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:17 smithi070 ceph-mon[26293]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:17.943 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:17 smithi070 ceph-mon[21827]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:18.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:17 smithi156 ceph-mon[22679]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: cluster: 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: id: be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: health: HEALTH_OK 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: services: 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: mgr: y(active, since 5m), standbys: x 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: data: 2024-04-24T16:25:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: pools: 6 pools, 161 pgs 2024-04-24T16:25:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: objects: 209 objects, 585 KiB 2024-04-24T16:25:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: usage: 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:25:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: pgs: 161 active+clean 2024-04-24T16:25:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:25:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: io: 2024-04-24T16:25:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-04-24T16:25:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:25:19.475 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:25:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:19] "GET /metrics HTTP/1.1" 200 207875 "" "Prometheus/2.33.4" 2024-04-24T16:25:19.498 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ls' 2024-04-24T16:25:20.088 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:19 smithi070 ceph-mon[26293]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:20.088 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:19 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3686008881' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:25:20.088 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:19 smithi070 ceph-mon[21827]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:20.088 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:19 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3686008881' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:25:20.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:19 smithi156 ceph-mon[22679]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:20.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:19 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3686008881' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi070=a;count:1 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:grafana ?:3000 1/1 3m ago 4m smithi156=a;count:1 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:mgr 2/2 3m ago 8m smithi070=y;smithi156=x;count:2 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:mon 3/3 3m ago 8m smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b;count:3 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi070=a;smithi156=b;count:2 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:osd 8 3m ago - 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus ?:9095 1/1 3m ago 5m smithi156=a;count:1 2024-04-24T16:25:21.313 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-04-24T16:25:21.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:21 smithi070 ceph-mon[26293]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:21.966 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:21 smithi070 ceph-mon[21827]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:21.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-24T16:25:22.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:21 smithi156 ceph-mon[22679]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:23.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:22 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:23.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:22 smithi070 ceph-mon[26293]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:22 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:22 smithi070 ceph-mon[21827]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:23.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:25:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:25:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:22 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:22 smithi156 ceph-mon[22679]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:24.092 INFO:teuthology.orchestra.run.smithi070.stdout:Scheduled to redeploy mgr.x on host 'smithi156' 2024-04-24T16:25:24.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:23 smithi070 ceph-mon[21827]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:24.092 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:23 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1827189406' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:25:24.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:23 smithi070 ceph-mon[26293]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:24.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:23 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1827189406' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:25:24.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:23 smithi156 ceph-mon[22679]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:24.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:23 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1827189406' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:25:24.778 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps --refresh' 2024-04-24T16:25:25.317 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[26293]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:25.317 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[26293]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:25.317 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:25.317 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[26293]: Schedule redeploy daemon mgr.x 2024-04-24T16:25:25.317 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:25:25.318 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[21827]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:25.318 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[21827]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:25.318 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:25.318 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[21827]: Schedule redeploy daemon mgr.x 2024-04-24T16:25:25.318 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:25 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:25:25.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:25 smithi156 ceph-mon[22679]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:25.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:25 smithi156 ceph-mon[22679]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:25.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:25 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:25.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:25 smithi156 ceph-mon[22679]: Schedule redeploy daemon mgr.x 2024-04-24T16:25:25.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:25 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:25:26.812 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:25:26.812 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 3m ago 4m 16.7M - ba2b418f427c 771467ffa56d 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (3m) 3m ago 3m 49.5M - 8.3.5 dad864ee21e9 5a7c9a93bc9e 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (3m) 3m ago 3m 62.5M - 3.5 e1d6a67b021e e257fa811608 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443 running (8m) 3m ago 8m 410M - 17.2.0 e1d6a67b021e df793845cb73 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e cbef4307b27d 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (10m) 3m ago 10m 45.0M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (8m) 3m ago 8m 41.2M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (8m) 3m ago 8m 39.4M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (4m) 3m ago 4m 15.0M - 1dbe0e931976 57518b11b60b 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (4m) 3m ago 4m 16.5M - 1dbe0e931976 a60500a59625 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (8m) 3m ago 8m 49.6M 4016M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (7m) 3m ago 7m 51.0M 4016M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (7m) 3m ago 7m 44.3M 4016M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (7m) 3m ago 7m 44.2M 4016M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:25:26.813 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (6m) 3m ago 6m 45.3M 4272M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:25:26.814 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (6m) 3m ago 6m 44.7M 4272M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:25:26.814 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (5m) 3m ago 5m 42.1M 4272M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:25:26.814 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (5m) 3m ago 5m 46.5M 4272M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:25:26.814 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (3m) 3m ago 4m 41.5M - 514e6a882f6e 14c1603136ac 2024-04-24T16:25:26.814 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (3m) 3m ago 3m 69.1M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:25:26.814 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (3m) 3m ago 3m 68.8M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:25:27.398 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:27 smithi070 ceph-mon[26293]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:27.398 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:27 smithi070 ceph-mon[26293]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:27.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:27 smithi070 ceph-mon[21827]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:27.398 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:27 smithi070 ceph-mon[21827]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:27.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:27 smithi156 ceph-mon[22679]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:27.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:27 smithi156 ceph-mon[22679]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:25:27.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'sleep 180' 2024-04-24T16:25:29.658 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:25:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:29] "GET /metrics HTTP/1.1" 200 207875 "" "Prometheus/2.33.4" 2024-04-24T16:25:29.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:29 smithi156 ceph-mon[22679]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:30.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:29 smithi070 ceph-mon[21827]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:30.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:29 smithi070 ceph-mon[26293]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:32.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:31 smithi070 ceph-mon[26293]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:32.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:31 smithi070 ceph-mon[21827]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:32.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:31 smithi156 ceph-mon[22679]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:33.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:32 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:33.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:32 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:33.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:32 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:25:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:25:34.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:33 smithi070 ceph-mon[26293]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:34.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:33 smithi070 ceph-mon[21827]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:34.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:33 smithi156 ceph-mon[22679]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:36.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:35 smithi070 ceph-mon[26293]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:36.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:35 smithi070 ceph-mon[21827]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:36.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:35 smithi156 ceph-mon[22679]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:38.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:37 smithi070 ceph-mon[26293]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:38.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:37 smithi070 ceph-mon[21827]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:38.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:37 smithi156 ceph-mon[22679]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:39.672 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:25:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:39] "GET /metrics HTTP/1.1" 200 207867 "" "Prometheus/2.33.4" 2024-04-24T16:25:39.672 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:39 smithi070 ceph-mon[21827]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:39 smithi070 ceph-mon[26293]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:40.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:39 smithi156 ceph-mon[22679]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:42.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:41 smithi070 ceph-mon[26293]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:42.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:41 smithi070 ceph-mon[21827]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:42.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:41 smithi156 ceph-mon[22679]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:43.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:42 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:43.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:42 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:43.081 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:42 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:43.081 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:25:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:25:44.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:43 smithi070 ceph-mon[26293]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:43 smithi070 ceph-mon[21827]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:44.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:43 smithi156 ceph-mon[22679]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:46.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:45 smithi070 ceph-mon[26293]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:46.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:45 smithi070 ceph-mon[21827]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:46.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:45 smithi156 ceph-mon[22679]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:48.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:47 smithi070 ceph-mon[26293]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:48.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:47 smithi070 ceph-mon[21827]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:47 smithi156 ceph-mon[22679]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:49.687 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:25:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:49] "GET /metrics HTTP/1.1" 200 207868 "" "Prometheus/2.33.4" 2024-04-24T16:25:49.687 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:49 smithi070 ceph-mon[21827]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:50.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:49 smithi070 ceph-mon[26293]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:49 smithi156 ceph-mon[22679]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:52.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:51 smithi070 ceph-mon[26293]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:52.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:51 smithi070 ceph-mon[21827]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:51 smithi156 ceph-mon[22679]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:53.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:52 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:53.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:52 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:53.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:52 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:25:53.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:25:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:25:53.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:53 smithi070 ceph-mon[26293]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:53.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:53 smithi070 ceph-mon[21827]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:53.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:25:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:25:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:25:53.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:25:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:25:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:25:54.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:53 smithi156 ceph-mon[22679]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:25:56.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:55 smithi070 ceph-mon[21827]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:56.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:55 smithi070 ceph-mon[26293]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:56.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:55 smithi156 ceph-mon[22679]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:57.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:57 smithi156 ceph-mon[22679]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:57.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:57 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:57.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:57 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:25:57.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:57 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:25:57.464 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[26293]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:57.464 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:57.464 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:25:57.464 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:25:57.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[21827]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:25:57.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:25:57.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:25:57.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:57 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:25:59.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:25:59] "GET /metrics HTTP/1.1" 200 207868 "" "Prometheus/2.33.4" 2024-04-24T16:26:00.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:26:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:26:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:26:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:26:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:25:59 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:26:00.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:26:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:26:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:26:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:25:59 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:26:01.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:00 smithi070 ceph-mon[21827]: Deploying daemon mgr.x on smithi156 2024-04-24T16:26:01.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:00 smithi070 ceph-mon[26293]: Deploying daemon mgr.x on smithi156 2024-04-24T16:26:01.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:00 smithi156 ceph-mon[22679]: Deploying daemon mgr.x on smithi156 2024-04-24T16:26:02.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:01 smithi070 ceph-mon[26293]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:02.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:01 smithi070 ceph-mon[21827]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:02.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:01 smithi156 ceph-mon[22679]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:03.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:02 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:03.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:02 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:03.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:02 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:03.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:26:03.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:03 smithi070 ceph-mon[26293]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:03.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:03 smithi070 ceph-mon[21827]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:03.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:26:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:03.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:03.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:04.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:03 smithi156 ceph-mon[22679]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:06.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:05 smithi070 ceph-mon[26293]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:06.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:05 smithi070 ceph-mon[21827]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:06.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:05 smithi156 ceph-mon[22679]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:08.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:07 smithi070 ceph-mon[26293]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:08.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:07 smithi070 ceph-mon[21827]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:08.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:07 smithi156 ceph-mon[22679]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:09.679 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:26:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:09] "GET /metrics HTTP/1.1" 200 207876 "" "Prometheus/2.33.4" 2024-04-24T16:26:10.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:09 smithi070 ceph-mon[26293]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:10.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:09 smithi070 ceph-mon[21827]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:10.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:09 smithi156 ceph-mon[22679]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:12.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:11 smithi070 ceph-mon[21827]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:12.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:11 smithi070 ceph-mon[26293]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:12.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:11 smithi156 ceph-mon[22679]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:13.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:12 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:13.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:12 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:13.058 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:26:13.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:12 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:13.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:13 smithi070 ceph-mon[26293]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:13 smithi070 ceph-mon[21827]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:26:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:14.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:13 smithi156 ceph-mon[22679]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:16.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:15 smithi070 ceph-mon[21827]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:16.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:15 smithi070 ceph-mon[26293]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:15 smithi156 ceph-mon[22679]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:17.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:26:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:26:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:26:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:26:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:26:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:26:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:26:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:26:17.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:26:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:26:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:26:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:26:18.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:17 smithi070 ceph-mon[26293]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:18.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:17 smithi070 ceph-mon[21827]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:18.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:17 smithi156 ceph-mon[22679]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:19.697 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:26:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:19] "GET /metrics HTTP/1.1" 200 207882 "" "Prometheus/2.33.4" 2024-04-24T16:26:20.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:19 smithi070 ceph-mon[21827]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:20.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:19 smithi070 ceph-mon[26293]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:20.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:19 smithi156 ceph-mon[22679]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:22.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:21 smithi070 ceph-mon[21827]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:22.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:21 smithi070 ceph-mon[26293]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:22.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:21 smithi156 ceph-mon[22679]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:23.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:22 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:22 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:23.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:22 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:23.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[23853]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:26:23.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:23 smithi070 ceph-mon[26293]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:23.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:23 smithi070 ceph-mon[21827]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:23.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:26:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:23.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:23.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:24.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:23 smithi156 ceph-mon[22679]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:26.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:25 smithi070 ceph-mon[26293]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:26.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:25 smithi070 ceph-mon[21827]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:26.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:25 smithi156 ceph-mon[22679]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:28.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:27 smithi070 ceph-mon[26293]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:28.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:27 smithi070 ceph-mon[21827]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:28.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:27 smithi156 ceph-mon[22679]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:29.711 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:26:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:29] "GET /metrics HTTP/1.1" 200 207882 "" "Prometheus/2.33.4" 2024-04-24T16:26:30.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:29 smithi070 ceph-mon[26293]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:30.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:29 smithi070 ceph-mon[21827]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:29 smithi156 ceph-mon[22679]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:32.031 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:31 smithi156 ceph-mon[22679]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:32.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:31 smithi070 ceph-mon[21827]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:32.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:31 smithi070 ceph-mon[26293]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:32.407 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 systemd[1]: Stopping Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:26:32.408 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 bash[42724]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-24T16:26:32.408 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 podman[42737]: 2024-04-24 16:26:32.20705019 +0000 UTC m=+0.138991209 container died df793845cb73617dd7f16f9b9acf9e799c824c1da182e2666579aa81cefcfdee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, version=8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-24T16:26:32.867 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:32 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:32.868 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 podman[42737]: 2024-04-24 16:26:32.569682766 +0000 UTC m=+0.501623775 container cleanup df793845cb73617dd7f16f9b9acf9e799c824c1da182e2666579aa81cefcfdee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, version=8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-24T16:26:32.868 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 bash[42737]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x 2024-04-24T16:26:32.868 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 bash[42763]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-24T16:26:33.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:32 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:33.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:32 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:33.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 podman[42751]: 2024-04-24 16:26:32.870966109 +0000 UTC m=+0.655893393 container remove df793845cb73617dd7f16f9b9acf9e799c824c1da182e2666579aa81cefcfdee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-type=git, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , architecture=x86_64, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container) 2024-04-24T16:26:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:26:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-24T16:26:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 systemd[1]: Stopped Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:26:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:32 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service: Consumed 17.345s CPU time. 2024-04-24T16:26:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 systemd[1]: Starting Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:26:33.547 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 podman[42871]: 2024-04-24 16:26:33.186049578 +0000 UTC m=+0.018174531 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:26:33.548 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 podman[42871]: 2024-04-24 16:26:33.28932614 +0000 UTC m=+0.121451134 container create 432b981bb958d0a8730b4645c5cad0b9142cfd8c386eebc00982d477c0d4d7ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-24T16:26:33.548 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 podman[42871]: 2024-04-24 16:26:33.474903226 +0000 UTC m=+0.307028179 container init 432b981bb958d0a8730b4645c5cad0b9142cfd8c386eebc00982d477c0d4d7ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-24T16:26:33.548 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 podman[42871]: 2024-04-24 16:26:33.477404471 +0000 UTC m=+0.309529429 container start 432b981bb958d0a8730b4645c5cad0b9142cfd8c386eebc00982d477c0d4d7ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:26:33.548 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-mgr[42890]: -- 172.21.15.156:0/505640086 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557f3fbb25a0 con 0x557f3fb87000 2024-04-24T16:26:33.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[26293]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:33.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:33.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:33.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:26:33.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:26:33.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:26:33.792 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:26:33.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:33.792 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:26:33.792 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:33.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[21827]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:33.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:33.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:33.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:26:33.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:26:33.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:33 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:26:33.835 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-mon[22679]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 bash[42871]: 432b981bb958d0a8730b4645c5cad0b9142cfd8c386eebc00982d477c0d4d7ea 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 systemd[1]: Started Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:33.635+0000 7f861cb0f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:26:33.836 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:33.837+0000 7f861cb0f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:26:34.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:33.950+0000 7f861cb0f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:26:34.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:34.289+0000 7f861cb0f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:26:34.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:34.375+0000 7f861cb0f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:26:35.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:34 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:35.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:34 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:35.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:34 smithi156 ceph-mon[22679]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:35.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:34 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:35.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:34 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:35.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:34 smithi070 ceph-mon[26293]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:34 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:34 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:35.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:34 smithi070 ceph-mon[21827]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:35.907 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:35.638+0000 7f861cb0f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:26:35.907 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:35.832+0000 7f861cb0f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:26:36.327 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:36.017+0000 7f861cb0f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:26:36.327 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:36.328+0000 7f861cb0f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:26:36.656 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:36.428+0000 7f861cb0f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:26:36.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:36.656+0000 7f861cb0f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:26:36.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:36.863+0000 7f861cb0f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:26:37.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:36 smithi070 ceph-mon[26293]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:37.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:36 smithi070 ceph-mon[21827]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:37.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:36 smithi156 ceph-mon[22679]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:37.783 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:37 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:37.491+0000 7f861cb0f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:26:38.038 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:37 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:37.933+0000 7f861cb0f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:26:38.038 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:38.039+0000 7f861cb0f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:26:38.351 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:38.259+0000 7f861cb0f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:26:39.049 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:38.797+0000 7f861cb0f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:26:39.049 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:38.892+0000 7f861cb0f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:26:39.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:39 smithi070 ceph-mon[21827]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:39.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:39 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:39.329 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:39 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:39.330 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:39 smithi070 ceph-mon[26293]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:39.330 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:39 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:39.330 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:39 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:39.368 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-mon[22679]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:39.368 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:39.368 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:26:39.368 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:39.138+0000 7f861cb0f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:26:39.368 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:39.272+0000 7f861cb0f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:26:39.368 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:39.369+0000 7f861cb0f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:26:39.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:39.489+0000 7f861cb0f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:26:39.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:26:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:39] "GET /metrics HTTP/1.1" 200 207895 "" "Prometheus/2.33.4" 2024-04-24T16:26:40.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:39 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:39.797+0000 7f861cb0f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:26:40.743 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:40.413+0000 7f861cb0f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:26:40.744 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:40.638+0000 7f861cb0f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:26:40.744 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:26:40] ENGINE Bus STARTING 2024-04-24T16:26:40.744 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CherryPy Checker: 2024-04-24T16:26:41.051 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: The Application mounted at '' has an empty config. 2024-04-24T16:26:41.051 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:26:41.051 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:26:40] ENGINE Serving on http://:::9283 2024-04-24T16:26:41.051 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:26:40] ENGINE Bus STARTED 2024-04-24T16:26:41.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:41 smithi156 ceph-mon[22679]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:41 smithi156 ceph-mon[22679]: Standby manager daemon x restarted 2024-04-24T16:26:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:41 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:26:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:41 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:26:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:41 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:26:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:41 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:26:41.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:41 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:26:41.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[21827]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[21827]: Standby manager daemon x restarted 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[26293]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[26293]: Standby manager daemon x restarted 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:26:41.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:26:41.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:26:41.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:26:41.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:41 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:26:42.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:42 smithi156 ceph-mon[22679]: mgrmap e21: y(active, since 6m), standbys: x 2024-04-24T16:26:42.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:42 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:42.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:42 smithi070 ceph-mon[21827]: mgrmap e21: y(active, since 6m), standbys: x 2024-04-24T16:26:42.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:42 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:42 smithi070 ceph-mon[26293]: mgrmap e21: y(active, since 6m), standbys: x 2024-04-24T16:26:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:42 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:43.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:26:43.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:43 smithi070 ceph-mon[21827]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:43.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:43 smithi070 ceph-mon[26293]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:43.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:26:43.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:43.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:43.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:43.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:43.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:44.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:43 smithi156 ceph-mon[22679]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:46.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:45 smithi070 ceph-mon[21827]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:46.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:45 smithi070 ceph-mon[26293]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:46.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:45 smithi156 ceph-mon[22679]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:48.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:47 smithi070 ceph-mon[26293]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:48.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:47 smithi070 ceph-mon[21827]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:47 smithi156 ceph-mon[22679]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:49.679 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:26:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:49] "GET /metrics HTTP/1.1" 200 207887 "" "Prometheus/2.33.4" 2024-04-24T16:26:50.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:49 smithi070 ceph-mon[26293]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:50.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:49 smithi070 ceph-mon[21827]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:49 smithi156 ceph-mon[22679]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:52.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:51 smithi070 ceph-mon[26293]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:51 smithi070 ceph-mon[21827]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:51 smithi156 ceph-mon[22679]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:53.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:52 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:53.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:52 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:53.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:26:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:26:53.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:52 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:26:53.761 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:53 smithi070 ceph-mon[21827]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:53.761 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:53 smithi070 ceph-mon[26293]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:53.762 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:26:53.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:53.762 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:26:53.762 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:26:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:26:53.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:26:54.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:53 smithi156 ceph-mon[22679]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:26:56.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:55 smithi070 ceph-mon[26293]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:56.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:55 smithi070 ceph-mon[21827]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:56.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:55 smithi156 ceph-mon[22679]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:58.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:57 smithi070 ceph-mon[26293]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:58.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:57 smithi070 ceph-mon[21827]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:58.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:57 smithi156 ceph-mon[22679]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:26:59.697 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:26:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:26:59] "GET /metrics HTTP/1.1" 200 207887 "" "Prometheus/2.33.4" 2024-04-24T16:27:00.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:26:59 smithi070 ceph-mon[26293]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:00.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:26:59 smithi070 ceph-mon[21827]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:00.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:26:59 smithi156 ceph-mon[22679]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:02.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:01 smithi070 ceph-mon[26293]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:02.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:01 smithi070 ceph-mon[21827]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:02.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:01 smithi156 ceph-mon[22679]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:03.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:02 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:03.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:02 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:03.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:27:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:27:03.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:02 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:03.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:03 smithi070 ceph-mon[21827]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:03.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:03 smithi070 ceph-mon[26293]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:03.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:27:03.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:03.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:03.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:03.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:03.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:04.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:03 smithi156 ceph-mon[22679]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:06.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:05 smithi070 ceph-mon[21827]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:06.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:05 smithi070 ceph-mon[26293]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:06.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:05 smithi156 ceph-mon[22679]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:08.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:07 smithi070 ceph-mon[21827]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:08.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:07 smithi070 ceph-mon[26293]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:08.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:07 smithi156 ceph-mon[22679]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:09.714 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:27:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:09] "GET /metrics HTTP/1.1" 200 207884 "" "Prometheus/2.33.4" 2024-04-24T16:27:10.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:09 smithi070 ceph-mon[26293]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:10.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:09 smithi070 ceph-mon[21827]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:10.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:09 smithi156 ceph-mon[22679]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:12.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:11 smithi070 ceph-mon[26293]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:12.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:11 smithi070 ceph-mon[21827]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:12.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:11 smithi156 ceph-mon[22679]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:13.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:12 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:13.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:12 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:13.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:12 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:13.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:27:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:27:13.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:13 smithi070 ceph-mon[26293]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:13 smithi070 ceph-mon[21827]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:27:13.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:13.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:14.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:13 smithi156 ceph-mon[22679]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:16.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:15 smithi070 ceph-mon[26293]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:16.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:15 smithi070 ceph-mon[21827]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:15 smithi156 ceph-mon[22679]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:17.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:27:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:27:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:27:17.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:27:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:27:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:27:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:27:17.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:27:17.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:27:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:27:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:27:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:27:18.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:17 smithi070 ceph-mon[26293]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:18.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:17 smithi070 ceph-mon[21827]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:18.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:17 smithi156 ceph-mon[22679]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:19.739 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:27:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:19] "GET /metrics HTTP/1.1" 200 207893 "" "Prometheus/2.33.4" 2024-04-24T16:27:20.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:19 smithi070 ceph-mon[26293]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:20.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:19 smithi070 ceph-mon[21827]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:20.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:19 smithi156 ceph-mon[22679]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:22.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:21 smithi070 ceph-mon[26293]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:22.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:21 smithi070 ceph-mon[21827]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:22.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:21 smithi156 ceph-mon[22679]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:23.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:22 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:23.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:22 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:22 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:23.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:27:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:27:23.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:23 smithi070 ceph-mon[21827]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:23.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:23 smithi070 ceph-mon[26293]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:23.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:27:23.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:23.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:23.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:23.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:24.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:23 smithi156 ceph-mon[22679]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:26.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:25 smithi070 ceph-mon[21827]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:26.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:25 smithi070 ceph-mon[26293]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:26.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:25 smithi156 ceph-mon[22679]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:28.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:27 smithi070 ceph-mon[21827]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:28.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:27 smithi070 ceph-mon[26293]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:28.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:27 smithi156 ceph-mon[22679]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:29.755 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:27:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:29] "GET /metrics HTTP/1.1" 200 207893 "" "Prometheus/2.33.4" 2024-04-24T16:27:30.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:29 smithi070 ceph-mon[21827]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:30.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:29 smithi070 ceph-mon[26293]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:30.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:29 smithi156 ceph-mon[22679]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:32.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:31 smithi070 ceph-mon[21827]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:32.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:31 smithi070 ceph-mon[26293]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:32.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:31 smithi156 ceph-mon[22679]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:33.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:32 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:33.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:32 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:33.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:32 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:27:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:27:33.763 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:27:33.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:33.763 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:33.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:33.763 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:33.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:34.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:33 smithi070 ceph-mon[26293]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:34.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:33 smithi070 ceph-mon[21827]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:34.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:33 smithi156 ceph-mon[22679]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:36.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:35 smithi070 ceph-mon[26293]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:36.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:35 smithi070 ceph-mon[21827]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:36.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:35 smithi156 ceph-mon[22679]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:38.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:37 smithi070 ceph-mon[26293]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:38.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:37 smithi070 ceph-mon[21827]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:38.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:37 smithi156 ceph-mon[22679]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:39.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:27:39.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:27:39.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:27:39.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:27:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:27:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:27:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:27:39.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:38 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:27:39.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:38 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:27:39.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:38 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:27:39.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:38 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:27:39.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:38 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:27:39.772 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:27:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:39] "GET /metrics HTTP/1.1" 200 207896 "" "Prometheus/2.33.4" 2024-04-24T16:27:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:39 smithi070 ceph-mon[26293]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:39 smithi070 ceph-mon[21827]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:40.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:39 smithi156 ceph-mon[22679]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:42.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:41 smithi070 ceph-mon[26293]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:42.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:41 smithi070 ceph-mon[21827]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:42.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:41 smithi156 ceph-mon[22679]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:43.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:42 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:43.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:42 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:43.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:42 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:43.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:27:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:27:43.781 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:27:43.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:43.782 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:43.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:43.782 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:44.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:43 smithi070 ceph-mon[26293]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:44.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:43 smithi070 ceph-mon[21827]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:44.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:43 smithi156 ceph-mon[22679]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:46.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:45 smithi070 ceph-mon[26293]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:46.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:45 smithi070 ceph-mon[21827]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:46.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:45 smithi156 ceph-mon[22679]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:48.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:47 smithi070 ceph-mon[26293]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:48.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:47 smithi070 ceph-mon[21827]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:47 smithi156 ceph-mon[22679]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:49.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:27:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:49] "GET /metrics HTTP/1.1" 200 207887 "" "Prometheus/2.33.4" 2024-04-24T16:27:50.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:49 smithi070 ceph-mon[21827]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:50.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:49 smithi070 ceph-mon[26293]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:49 smithi156 ceph-mon[22679]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:51 smithi156 ceph-mon[22679]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:52.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:51 smithi070 ceph-mon[21827]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:52.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:51 smithi070 ceph-mon[26293]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:53.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:52 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:53.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:27:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:27:53.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:52 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:53.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:52 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:27:53.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:27:53.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:53.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:53.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:27:53.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:27:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:27:53.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:27:54.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:53 smithi156 ceph-mon[22679]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:54.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:53 smithi070 ceph-mon[26293]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:54.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:53 smithi070 ceph-mon[21827]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:27:56.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:55 smithi156 ceph-mon[22679]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:56.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:55 smithi070 ceph-mon[26293]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:56.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:55 smithi070 ceph-mon[21827]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:58.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:57 smithi156 ceph-mon[22679]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:57 smithi070 ceph-mon[26293]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:58.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:57 smithi070 ceph-mon[21827]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:27:59.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:27:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:27:59] "GET /metrics HTTP/1.1" 200 207887 "" "Prometheus/2.33.4" 2024-04-24T16:28:00.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:27:59 smithi156 ceph-mon[22679]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:00.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:27:59 smithi070 ceph-mon[26293]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:00.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:27:59 smithi070 ceph-mon[21827]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:02.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:01 smithi156 ceph-mon[22679]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:02.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:01 smithi070 ceph-mon[26293]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:02.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:01 smithi070 ceph-mon[21827]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:03.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:02 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:03.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:28:03.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:02 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:03.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:02 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:03.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:28:03.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:03.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:03.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:28:03.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:03.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:04.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:03 smithi156 ceph-mon[22679]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:04.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:03 smithi070 ceph-mon[26293]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:04.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:03 smithi070 ceph-mon[21827]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:06.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:05 smithi156 ceph-mon[22679]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:06.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:05 smithi070 ceph-mon[26293]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:06.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:05 smithi070 ceph-mon[21827]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:08.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:07 smithi156 ceph-mon[22679]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:08.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:07 smithi070 ceph-mon[26293]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:08.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:07 smithi070 ceph-mon[21827]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:09.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:09] "GET /metrics HTTP/1.1" 200 207888 "" "Prometheus/2.33.4" 2024-04-24T16:28:10.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:09 smithi156 ceph-mon[22679]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:10.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:09 smithi070 ceph-mon[26293]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:10.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:09 smithi070 ceph-mon[21827]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:12.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:11 smithi156 ceph-mon[22679]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:12.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:11 smithi070 ceph-mon[26293]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:12.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:11 smithi070 ceph-mon[21827]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:13.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:12 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:13.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:28:13.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:12 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:13.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:12 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:13.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:28:13.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:13.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:28:13.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:13.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:14.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:13 smithi156 ceph-mon[22679]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:14.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:13 smithi070 ceph-mon[26293]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:14.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:13 smithi070 ceph-mon[21827]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:15 smithi156 ceph-mon[22679]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:16.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:15 smithi070 ceph-mon[26293]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:16.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:15 smithi070 ceph-mon[21827]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:17.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:16 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:16 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:17.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:17.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:17.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:17.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:17.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:16 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:18.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:17 smithi156 ceph-mon[22679]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:18.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:17 smithi070 ceph-mon[26293]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:18.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:17 smithi070 ceph-mon[21827]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:19.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:19] "GET /metrics HTTP/1.1" 200 207890 "" "Prometheus/2.33.4" 2024-04-24T16:28:20.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:19 smithi156 ceph-mon[22679]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:20.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:19 smithi070 ceph-mon[26293]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:19 smithi070 ceph-mon[21827]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:22.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:21 smithi156 ceph-mon[22679]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:22.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:21 smithi070 ceph-mon[21827]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:22.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:21 smithi070 ceph-mon[26293]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:23.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:28:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:22 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:23.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:22 smithi156 ceph-mon[22679]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:23.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:22 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:23.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:22 smithi070 ceph-mon[26293]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:22 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:23.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:22 smithi070 ceph-mon[21827]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:23.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:28:23.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:28:23.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:23.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:28:23.790 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:25.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:24 smithi070 ceph-mon[21827]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:25.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:24 smithi070 ceph-mon[26293]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:25.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:24 smithi156 ceph-mon[22679]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:27.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:27 smithi070 ceph-mon[26293]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:27.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:27 smithi070 ceph-mon[21827]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:27.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:27 smithi156 ceph-mon[22679]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:29.698 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:29] "GET /metrics HTTP/1.1" 200 207890 "" "Prometheus/2.33.4" 2024-04-24T16:28:29.771 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:28:30.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:29 smithi070 ceph-mon[26293]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:30.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:29 smithi070 ceph-mon[21827]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:30.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:29 smithi156 ceph-mon[22679]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (6m) 2m ago 7m 26.2M - ba2b418f427c 771467ffa56d 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (6m) 113s ago 6m 55.2M - 8.3.5 dad864ee21e9 5a7c9a93bc9e 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (6m) 2m ago 6m 49.0M - 3.5 e1d6a67b021e e257fa811608 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283 running (118s) 113s ago 11m 64.0M - 19.0.0-3285-g5904d294 8bb2087275ff 432b981bb958 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:9283 running (13m) 2m ago 13m 460M - 17.2.0 e1d6a67b021e cbef4307b27d 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (13m) 2m ago 13m 48.2M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (11m) 113s ago 11m 42.1M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:28:31.704 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (11m) 2m ago 11m 39.5M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (7m) 2m ago 7m 16.5M - 1dbe0e931976 57518b11b60b 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (7m) 113s ago 7m 18.2M - 1dbe0e931976 a60500a59625 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (11m) 2m ago 11m 52.0M 4016M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (10m) 2m ago 10m 53.0M 4016M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (10m) 2m ago 10m 47.1M 4016M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (10m) 2m ago 10m 47.6M 4016M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (9m) 113s ago 9m 48.6M 4272M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (9m) 113s ago 9m 48.0M 4272M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (8m) 113s ago 8m 45.3M 4272M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (8m) 113s ago 8m 50.0M 4272M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (6m) 113s ago 7m 66.3M - 514e6a882f6e 14c1603136ac 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (6m) 2m ago 6m 70.2M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:28:31.705 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (6m) 113s ago 6m 70.0M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:28:32.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:31 smithi070 ceph-mon[21827]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:32.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:31 smithi070 ceph-mon[26293]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:32.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:31 smithi156 ceph-mon[22679]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:32.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions' 2024-04-24T16:28:32.870 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:32 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:32.871 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:32 smithi070 ceph-mon[21827]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:28:32.871 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:32 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:32.871 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:32 smithi070 ceph-mon[26293]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:28:33.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:32 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:33.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:32 smithi156 ceph-mon[22679]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:28:33.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:28:33.635 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:28:33.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:33.635 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:33.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:28:33.635 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:33.902 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:33 smithi070 ceph-mon[26293]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:33.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:33 smithi070 ceph-mon[21827]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:34.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:33 smithi156 ceph-mon[22679]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 1 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:28:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 1 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:28:34.245 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:28:34.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:34 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3212719215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:28:34.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:34 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3212719215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:28:34.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph -s' 2024-04-24T16:28:35.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:34 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3212719215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:28:36.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:35 smithi070 ceph-mon[26293]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:36.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:35 smithi070 ceph-mon[21827]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:36.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:35 smithi156 ceph-mon[22679]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:36.907 INFO:teuthology.orchestra.run.smithi070.stdout: cluster: 2024-04-24T16:28:36.907 INFO:teuthology.orchestra.run.smithi070.stdout: id: be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: health: HEALTH_OK 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: services: 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: mon: 3 daemons, quorum a,c,b (age 11m) 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: mgr: y(active, since 8m), standbys: x 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 8m) 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: data: 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: pools: 6 pools, 161 pgs 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: objects: 209 objects, 585 KiB 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: usage: 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: pgs: 161 active+clean 2024-04-24T16:28:36.908 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:28:36.909 INFO:teuthology.orchestra.run.smithi070.stdout: io: 2024-04-24T16:28:36.909 INFO:teuthology.orchestra.run.smithi070.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-24T16:28:36.909 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:28:37.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:28:38.033 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:37 smithi070 ceph-mon[26293]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:38.034 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:37 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3236327871' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:28:38.034 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:37 smithi070 ceph-mon[21827]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:38.034 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:37 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3236327871' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:28:38.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:37 smithi156 ceph-mon[22679]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:38.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:37 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3236327871' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:28:38.895 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:38 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:28:38.895 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:38 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:28:38.895 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:38 smithi070 ceph-mon[26293]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:28:38.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:38 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:28:38.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:38 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:28:38.895 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:38 smithi070 ceph-mon[21827]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:28:38.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:38 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:28:38.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:38 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:28:38.980 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:38 smithi156 ceph-mon[22679]: from='mgr.24328 172.21.15.70:0/474497153' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:28:39.352 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_OK 2024-04-24T16:28:39.717 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:39] "GET /metrics HTTP/1.1" 200 207887 "" "Prometheus/2.33.4" 2024-04-24T16:28:39.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-04-24T16:28:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:39 smithi070 ceph-mon[26293]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:39 smithi070 ceph-mon[26293]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:28:40.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:39 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2927618530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:28:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:39 smithi070 ceph-mon[21827]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:39 smithi070 ceph-mon[21827]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:28:40.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:39 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2927618530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:28:40.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:39 smithi156 ceph-mon[22679]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:40.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:39 smithi156 ceph-mon[22679]: from='mgr.24328 ' entity='mgr.y' 2024-04-24T16:28:40.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:39 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2927618530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:28:41.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:41 smithi070 ceph-mon[26293]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:41.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:41 smithi070 ceph-mon[21827]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:41.901 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:28:42.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:41 smithi156 ceph-mon[22679]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:28:42.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph mgr fail' 2024-04-24T16:28:42.979 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:42 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:42.979 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:42 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/454505991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:28:42.980 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:42 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:42.980 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:42 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/454505991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:28:43.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:42 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:43.059 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:42 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/454505991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:28:43.059 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:28:43.616 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:28:43.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:43.617 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:28:43.617 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:43.873 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:43 smithi070 ceph-mon[26293]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:43.873 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:43 smithi070 ceph-mon[21827]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:44.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:43 smithi156 ceph-mon[22679]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:28:45.011 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/738412431' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:28:45.011 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-mon[26293]: osdmap e95: 8 total, 8 up, 8 in 2024-04-24T16:28:45.011 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:44.787Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:28:45.011 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:44.735+0000 7fe1fbdce700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-24T16:28:45.012 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setuser ceph since I am not root 2024-04-24T16:28:45.012 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ignoring --setgroup ceph since I am not root 2024-04-24T16:28:45.012 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-mgr[22061]: -- 172.21.15.70:0/1779152074 <== mon.1 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562ae37ae340 con 0x562ae3786800 2024-04-24T16:28:45.012 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:44.915+0000 7ff1a2ff9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:28:45.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/738412431' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:28:45.012 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:44 smithi070 ceph-mon[21827]: osdmap e95: 8 total, 8 up, 8 in 2024-04-24T16:28:45.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/738412431' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:28:45.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-mon[22679]: osdmap e95: 8 total, 8 up, 8 in 2024-04-24T16:28:45.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:28:44] ENGINE Bus STOPPING 2024-04-24T16:28:45.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:28:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:28:45.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:28:44] ENGINE Bus STOPPED 2024-04-24T16:28:45.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:28:44] ENGINE Bus STARTING 2024-04-24T16:28:45.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:28:44] ENGINE Serving on http://:::9283 2024-04-24T16:28:45.158 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:44 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:28:44] ENGINE Bus STARTED 2024-04-24T16:28:45.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:45.059+0000 7ff1a2ff9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:28:45.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:45.161+0000 7ff1a2ff9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:28:45.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'sleep 180' 2024-04-24T16:28:45.731 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:45.492+0000 7ff1a2ff9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:28:45.731 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:45.579+0000 7ff1a2ff9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:28:45.731 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:45.431Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:28:46.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/738412431' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:28:46.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: mgrmap e22: x(active, starting, since 0.393274s) 2024-04-24T16:28:46.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:28:46.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:28:46.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:28:46.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:28:46.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:28:46.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:28:46.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:28:46.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:28:46.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:28:46.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:28:46.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:28:46.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:28:46.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:28:46.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:28:46.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:28:46.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: Manager daemon x is now available 2024-04-24T16:28:46.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: Queued rgw.foo for migration 2024-04-24T16:28:46.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-24T16:28:46.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:28:46.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:28:46.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:46.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:46.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:46.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:46.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: Deploying cephadm binary to smithi156 2024-04-24T16:28:46.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: Deploying cephadm binary to smithi070 2024-04-24T16:28:46.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/738412431' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:28:46.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: mgrmap e22: x(active, starting, since 0.393274s) 2024-04-24T16:28:46.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:28:46.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:28:46.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:28:46.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:28:46.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:28:46.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:28:46.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:28:46.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:28:46.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:28:46.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:28:46.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:28:46.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:28:46.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:28:46.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:28:46.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:28:46.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: Manager daemon x is now available 2024-04-24T16:28:46.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: Queued rgw.foo for migration 2024-04-24T16:28:46.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-24T16:28:46.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:28:46.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:28:46.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:46.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:46.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:46.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:46.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: Deploying cephadm binary to smithi156 2024-04-24T16:28:46.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: Deploying cephadm binary to smithi070 2024-04-24T16:28:46.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:45 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/738412431' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:28:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: mgrmap e22: x(active, starting, since 0.393274s) 2024-04-24T16:28:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:28:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:28:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:28:46.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: Manager daemon x is now available 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: Queued rgw.foo for migration 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-24T16:28:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: Deploying cephadm binary to smithi156 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: Deploying cephadm binary to smithi070 2024-04-24T16:28:46.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:45 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:46.340 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:46.276Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:46.911 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:46.741+0000 7ff1a2ff9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:28:46.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[21827]: mgrmap e23: x(active, since 1.39923s) 2024-04-24T16:28:46.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[21827]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:46.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[21827]: [24/Apr/2024:16:28:45] ENGINE Bus STARTING 2024-04-24T16:28:46.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[21827]: [24/Apr/2024:16:28:46] ENGINE Serving on https://172.21.15.156:7150 2024-04-24T16:28:46.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[21827]: [24/Apr/2024:16:28:46] ENGINE Serving on http://172.21.15.156:8765 2024-04-24T16:28:46.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[21827]: [24/Apr/2024:16:28:46] ENGINE Bus STARTED 2024-04-24T16:28:46.912 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[26293]: mgrmap e23: x(active, since 1.39923s) 2024-04-24T16:28:46.913 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[26293]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:46.913 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[26293]: [24/Apr/2024:16:28:45] ENGINE Bus STARTING 2024-04-24T16:28:46.913 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[26293]: [24/Apr/2024:16:28:46] ENGINE Serving on https://172.21.15.156:7150 2024-04-24T16:28:46.913 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[26293]: [24/Apr/2024:16:28:46] ENGINE Serving on http://172.21.15.156:8765 2024-04-24T16:28:46.913 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-mon[26293]: [24/Apr/2024:16:28:46] ENGINE Bus STARTED 2024-04-24T16:28:47.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:46 smithi156 ceph-mon[22679]: mgrmap e23: x(active, since 1.39923s) 2024-04-24T16:28:47.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:46 smithi156 ceph-mon[22679]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:47.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:46 smithi156 ceph-mon[22679]: [24/Apr/2024:16:28:45] ENGINE Bus STARTING 2024-04-24T16:28:47.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:46 smithi156 ceph-mon[22679]: [24/Apr/2024:16:28:46] ENGINE Serving on https://172.21.15.156:7150 2024-04-24T16:28:47.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:46 smithi156 ceph-mon[22679]: [24/Apr/2024:16:28:46] ENGINE Serving on http://172.21.15.156:8765 2024-04-24T16:28:47.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:46 smithi156 ceph-mon[22679]: [24/Apr/2024:16:28:46] ENGINE Bus STARTED 2024-04-24T16:28:47.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:46.913+0000 7ff1a2ff9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:28:47.323 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:46.999+0000 7ff1a2ff9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:28:47.745 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:47.317+0000 7ff1a2ff9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:28:47.745 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:47.409+0000 7ff1a2ff9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:28:47.745 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:47.503+0000 7ff1a2ff9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:28:48.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-mon[26293]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:48.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-mon[26293]: mgrmap e24: x(active, since 2s) 2024-04-24T16:28:48.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-mon[21827]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:48.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-mon[21827]: mgrmap e24: x(active, since 2s) 2024-04-24T16:28:48.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:47.750+0000 7ff1a2ff9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:28:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:47 smithi156 ceph-mon[22679]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:47 smithi156 ceph-mon[22679]: mgrmap e24: x(active, since 2s) 2024-04-24T16:28:48.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:48.191+0000 7ff1a2ff9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:28:49.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:48.679+0000 7ff1a2ff9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:28:49.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:48 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:48.778+0000 7ff1a2ff9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:28:49.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:49.077+0000 7ff1a2ff9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:28:49.540 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:49.168+0000 7ff1a2ff9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:28:49.915 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-mon[26293]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:49.916 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-mon[26293]: mgrmap e25: x(active, since 4s) 2024-04-24T16:28:49.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-mon[21827]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:49.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-mon[21827]: mgrmap e25: x(active, since 4s) 2024-04-24T16:28:49.916 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:49.571+0000 7ff1a2ff9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:28:49.916 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:49.707+0000 7ff1a2ff9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:28:49.916 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:49.804+0000 7ff1a2ff9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:28:49.917 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:49 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:49.918+0000 7ff1a2ff9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:28:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:49 smithi156 ceph-mon[22679]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:49 smithi156 ceph-mon[22679]: mgrmap e25: x(active, since 4s) 2024-04-24T16:28:50.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:50.015+0000 7ff1a2ff9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:28:50.871 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:50.610+0000 7ff1a2ff9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:28:51.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:50.980+0000 7ff1a2ff9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:28:51.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:28:50] ENGINE Bus STARTING 2024-04-24T16:28:51.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: CherryPy Checker: 2024-04-24T16:28:51.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: The Application mounted at '' has an empty config. 2024-04-24T16:28:51.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: 2024-04-24T16:28:51.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:28:51] ENGINE Serving on http://:::9283 2024-04-24T16:28:51.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: [24/Apr/2024:16:28:51] ENGINE Bus STARTED 2024-04-24T16:28:52.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[21827]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:52.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[21827]: Standby manager daemon y started 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[26293]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[26293]: Standby manager daemon y started 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:28:52.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:28:52.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:51 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:52.156 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:51 smithi156 ceph-mon[22679]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:28:52.156 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:51 smithi156 ceph-mon[22679]: Standby manager daemon y started 2024-04-24T16:28:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:51 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:28:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:51 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:28:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:51 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:28:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:51 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/384882140' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:28:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:51 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:28:53.016 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:28:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:52] "GET /metrics HTTP/1.1" 200 34896 "" "Prometheus/2.33.4" 2024-04-24T16:28:53.017 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:52 smithi156 ceph-mon[22679]: mgrmap e26: x(active, since 7s), standbys: y 2024-04-24T16:28:53.017 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:52 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:28:53.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:52 smithi070 ceph-mon[21827]: mgrmap e26: x(active, since 7s), standbys: y 2024-04-24T16:28:53.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:52 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:28:53.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:52 smithi070 ceph-mon[26293]: mgrmap e26: x(active, since 7s), standbys: y 2024-04-24T16:28:53.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:52 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:28:53.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-mon[21827]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:28:53.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:53.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:53.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-mon[26293]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:28:53.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:53.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:53.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=error ts=2024-04-24T16:28:53.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.70:8443//api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:53.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.156:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.156 because it doesn't contain any IP SANs" 2024-04-24T16:28:53.791 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:28:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=warn ts=2024-04-24T16:28:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.70:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.70 because it doesn't contain any IP SANs" 2024-04-24T16:28:53.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:53 smithi156 ceph-mon[22679]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:28:53.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:53 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:53.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:53 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:55 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:55.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:55 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi070 to 2992M 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: Adjusting osd_memory_target on smithi156 to 3248M 2024-04-24T16:28:56.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi070 to 2992M 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: Adjusting osd_memory_target on smithi156 to 3248M 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:28:56.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:56 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:28:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi070 to 2992M 2024-04-24T16:28:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:28:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T16:28:56.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: Adjusting osd_memory_target on smithi156 to 3248M 2024-04-24T16:28:56.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:56.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:28:56.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:28:56.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:28:56.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:28:56.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:28:56.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:56 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:28:57.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:56 smithi070 systemd[1]: Stopping Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:28:57.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:56 smithi070 bash[55452]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-24T16:28:57.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 podman[55465]: 2024-04-24 16:28:57.036945767 +0000 UTC m=+0.109287292 container died 57518b11b60ba9590825cb5d13e502d2ed30dd34374e1acde851176c5319cef2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:28:57.641 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 podman[55465]: 2024-04-24 16:28:57.410354196 +0000 UTC m=+0.482695728 container cleanup 57518b11b60ba9590825cb5d13e502d2ed30dd34374e1acde851176c5319cef2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:28:57.641 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 bash[55465]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a 2024-04-24T16:28:57.641 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 bash[55492]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.642 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[26293]: Reconfiguring daemon node-exporter.a on smithi070 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.643 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-24T16:28:57.644 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:57 smithi070 ceph-mon[21827]: Reconfiguring daemon node-exporter.a on smithi070 2024-04-24T16:28:57.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:28:57.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:28:57.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-24T16:28:57.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:57 smithi156 ceph-mon[22679]: Reconfiguring daemon node-exporter.a on smithi070 2024-04-24T16:28:57.892 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 podman[55479]: 2024-04-24 16:28:57.644482057 +0000 UTC m=+0.611685637 container remove 57518b11b60ba9590825cb5d13e502d2ed30dd34374e1acde851176c5319cef2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:28:57.892 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:28:57.892 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-24T16:28:57.892 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 systemd[1]: Stopped Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:28:57.893 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 3.683s CPU time. 2024-04-24T16:28:57.893 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 systemd[1]: Starting Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:28:57.893 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 podman[55559]: 2024-04-24 16:28:57.773661917 +0000 UTC m=+0.017418747 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-24T16:28:57.893 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:57 smithi070 podman[55559]: 2024-04-24 16:28:57.895732646 +0000 UTC m=+0.139489451 container create a0dadc7082e4ef75ea03c8d85c0c31a2d7217a9bd049266dd15b22939921f755 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 podman[55559]: 2024-04-24 16:28:58.064738053 +0000 UTC m=+0.308494856 container init a0dadc7082e4ef75ea03c8d85c0c31a2d7217a9bd049266dd15b22939921f755 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 podman[55559]: 2024-04-24 16:28:58.067546365 +0000 UTC m=+0.311303171 container start a0dadc7082e4ef75ea03c8d85c0c31a2d7217a9bd049266dd15b22939921f755 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.070Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.071Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.071Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.071Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=arp 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-24T16:28:58.291 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=edac 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-24T16:28:58.292 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=os 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-24T16:28:58.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=stat 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=time 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-24T16:28:58.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=uname 2024-04-24T16:28:58.295 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-24T16:28:58.295 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-24T16:28:58.295 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-24T16:28:58.295 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-24T16:28:58.295 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[55573]: ts=2024-04-24T16:28:58.072Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-24T16:28:58.295 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 bash[55559]: a0dadc7082e4ef75ea03c8d85c0c31a2d7217a9bd049266dd15b22939921f755 2024-04-24T16:28:58.295 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:28:58 smithi070 systemd[1]: Started Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:28:58.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-mon[21827]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:28:58.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:58.759 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:58.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-mon[26293]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:28:58.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:58.760 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:58 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:58.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:58 smithi156 ceph-mon[22679]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:28:58.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:58 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:58.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:58 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:28:59.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:28:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:28:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:28:59.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:59 smithi070 ceph-mon[26293]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-24T16:28:59.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:28:59 smithi070 ceph-mon[26293]: Reconfiguring daemon alertmanager.a on smithi070 2024-04-24T16:28:59.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:59 smithi070 ceph-mon[21827]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-24T16:28:59.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:28:59 smithi070 ceph-mon[21827]: Reconfiguring daemon alertmanager.a on smithi070 2024-04-24T16:28:59.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:59 smithi156 ceph-mon[22679]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-24T16:28:59.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:28:59 smithi156 ceph-mon[22679]: Reconfiguring daemon alertmanager.a on smithi070 2024-04-24T16:29:00.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:00 smithi070 ceph-mon[26293]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:00.592 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:00 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:00.592 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:00 smithi070 ceph-mon[21827]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:00.592 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:00 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:00.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:00 smithi156 ceph-mon[22679]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:00.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:00 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:01.616 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:01 smithi070 systemd[1]: Stopping Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:01.616 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:01 smithi070 bash[55753]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-24T16:29:01.909 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[46258]: level=info ts=2024-04-24T16:29:01.619Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:29:01.909 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:01 smithi070 podman[55767]: 2024-04-24 16:29:01.639483259 +0000 UTC m=+0.128005752 container died 771467ffa56d57a1df2fe73b218566992ccbf654831211317dcab3c0c8cae2b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:02.290 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 podman[55767]: 2024-04-24 16:29:02.011355637 +0000 UTC m=+0.499878125 container cleanup 771467ffa56d57a1df2fe73b218566992ccbf654831211317dcab3c0c8cae2b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:02.290 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 bash[55767]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a 2024-04-24T16:29:02.290 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 bash[55795]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-24T16:29:02.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:02 smithi070 ceph-mon[26293]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:02.660 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:02 smithi070 ceph-mon[26293]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:02.661 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:02 smithi070 ceph-mon[21827]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:02.661 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:02 smithi070 ceph-mon[21827]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:02.661 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 podman[55781]: 2024-04-24 16:29:02.362525242 +0000 UTC m=+0.721154861 container remove 771467ffa56d57a1df2fe73b218566992ccbf654831211317dcab3c0c8cae2b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:02.661 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 podman[55781]: 2024-04-24 16:29:02.438304769 +0000 UTC m=+0.796934344 volume remove 8c4b1742d781d373289ff61936abee73491fb2a959c6ba8eb7bf3070e66d2a86 2024-04-24T16:29:02.661 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-24T16:29:02.661 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 systemd[1]: Stopped Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:02.661 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service: Consumed 1.030s CPU time. 2024-04-24T16:29:02.662 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 systemd[1]: Starting Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:02.683 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:02 smithi156 ceph-mon[22679]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:02.683 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:02 smithi156 ceph-mon[22679]: from='client.34589 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:02.976 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 podman[55862]: 2024-04-24 16:29:02.664055358 +0000 UTC m=+0.104686597 volume create e674ffab6f58e438eaf93c5d30af1f55e52fc391452dd13b3f59b1b8a41fda67 2024-04-24T16:29:02.976 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 podman[55862]: 2024-04-24 16:29:02.576978208 +0000 UTC m=+0.017609444 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-24T16:29:02.977 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:02 smithi070 podman[55862]: 2024-04-24 16:29:02.773016207 +0000 UTC m=+0.213647485 container create 609b591dc19cf80f9834141824ed44d8127ae4b6a9db6ed649674afeef3498bb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:03.157 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:29:02] "GET /metrics HTTP/1.1" 200 37886 "" "Prometheus/2.33.4" 2024-04-24T16:29:03.290 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 podman[55862]: 2024-04-24 16:29:03.092765728 +0000 UTC m=+0.533396964 container init 609b591dc19cf80f9834141824ed44d8127ae4b6a9db6ed649674afeef3498bb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:03.290 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 podman[55862]: 2024-04-24 16:29:03.095490769 +0000 UTC m=+0.536122006 container start 609b591dc19cf80f9834141824ed44d8127ae4b6a9db6ed649674afeef3498bb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:03.290 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.104Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-24T16:29:03.290 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.104Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.106Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.70 port=9094 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.107Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.131Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.132Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.134Z caller=main.go:518 msg=Listening address=:9093 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:03.134Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 bash[55862]: 609b591dc19cf80f9834141824ed44d8127ae4b6a9db6ed649674afeef3498bb 2024-04-24T16:29:03.291 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:03 smithi070 systemd[1]: Started Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:04.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:04.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:04.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:04.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:04.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:04 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:29:04.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:29:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:29:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:04.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:04 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:29:05.357 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:05 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:05.107Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000220644s 2024-04-24T16:29:06.492 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:06 smithi070 ceph-mon[26293]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:06.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:06 smithi070 ceph-mon[21827]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:06.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-mon[22679]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:29:06.597+0000 7f85a1eb6700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:29:06.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:06.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:29:06.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:06.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:06.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:08.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:08.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:08.545 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:08.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:08.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:08.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[26293]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[26293]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[26293]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:08.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:08.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[21827]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:29:08.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[21827]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:29:08.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:08 smithi070 ceph-mon[21827]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:29:08.563 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:08 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:08.563 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:29:08.564 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:08.565 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:08.565 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:08.565 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:08.565 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:08.565 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:08.565 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:08.565 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:29:08.566 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.567 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:08.568 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:08.569 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:08 smithi156 ceph-mon[22679]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:29:08.569 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:08 smithi156 ceph-mon[22679]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:29:08.569 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:08 smithi156 ceph-mon[22679]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:29:09.790 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:29:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[22057]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:29:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:29:10.157 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 systemd[1]: Stopping Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:10.157 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 bash[45925]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:29:10.520 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-mon[22679]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:10.520 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:29:10.520 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-24T16:29:10.520 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-24T16:29:10.520 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-24T16:29:10.520 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.205Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.206Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.206Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[39579]: ts=2024-04-24T16:29:10.206Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-24T16:29:10.521 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 podman[45938]: 2024-04-24 16:29:10.215482178 +0000 UTC m=+0.105720721 container died 14c1603136ac17e876390d9937f667eadb2f2e505af9ceee1151fba5deaa5ba2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:10.540 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:10 smithi070 ceph-mon[21827]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:10.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:10 smithi070 ceph-mon[26293]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:10.853 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 podman[45938]: 2024-04-24 16:29:10.613951357 +0000 UTC m=+0.504189902 container cleanup 14c1603136ac17e876390d9937f667eadb2f2e505af9ceee1151fba5deaa5ba2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:10.853 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 bash[45938]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a 2024-04-24T16:29:10.854 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 bash[45965]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:29:11.104 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 podman[45952]: 2024-04-24 16:29:10.857000936 +0000 UTC m=+0.643677975 container remove 14c1603136ac17e876390d9937f667eadb2f2e505af9ceee1151fba5deaa5ba2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:11.104 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-24T16:29:11.104 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 systemd[1]: Stopped Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:11.105 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 1.904s CPU time. 2024-04-24T16:29:11.105 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:10 smithi156 systemd[1]: Starting Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:11.105 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 podman[46034]: 2024-04-24 16:29:10.99108784 +0000 UTC m=+0.019598309 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-24T16:29:11.105 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 podman[46034]: 2024-04-24 16:29:11.107805467 +0000 UTC m=+0.136315923 container create 0e9a48e5d60edc83024f412c27eae570cdc3b7861526196dac61ba516a2256eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:11.368 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 podman[46034]: 2024-04-24 16:29:11.34453863 +0000 UTC m=+0.373049093 container init 0e9a48e5d60edc83024f412c27eae570cdc3b7861526196dac61ba516a2256eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:11.368 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 podman[46034]: 2024-04-24 16:29:11.347229551 +0000 UTC m=+0.375740015 container start 0e9a48e5d60edc83024f412c27eae570cdc3b7861526196dac61ba516a2256eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:29:11.657 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.371Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.371Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.371Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.371Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi156 (none))" 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.371Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.371Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.373Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.373Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.374Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.377Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-24T16:29:11.658 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.377Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.89µs 2024-04-24T16:29:11.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.377Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-24T16:29:11.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 bash[46034]: 0e9a48e5d60edc83024f412c27eae570cdc3b7861526196dac61ba516a2256eb 2024-04-24T16:29:11.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 systemd[1]: Started Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:11.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.593Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-24T16:29:12.154 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 systemd[1]: Stopping Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:12.154 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 bash[46159]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-24T16:29:12.154 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.831Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-24T16:29:12.154 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.831Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-24T16:29:12.154 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.831Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.457µs wal_replay_duration=454.618645ms total_replay_duration=454.668844ms 2024-04-24T16:29:12.154 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.836Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-24T16:29:12.155 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.836Z caller=main.go:947 level=info msg="TSDB started" 2024-04-24T16:29:12.155 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.836Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-24T16:29:12.155 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.870Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=33.596216ms db_storage=881ns remote_storage=1.58µs web_handler=397ns query_engine=725ns scrape=548.49µs scrape_sd=369.092µs notify=19.162µs notify_sd=16.612µs rules=32.110955ms 2024-04-24T16:29:12.155 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:11.870Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-24T16:29:12.407 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 podman[46169]: 2024-04-24 16:29:12.157369799 +0000 UTC m=+0.111274176 container died a60500a596252b835a3f352020a68eee6d03bb718b960c3d3fbc3378dc09314c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:29:12.703 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 podman[46169]: 2024-04-24 16:29:12.488735331 +0000 UTC m=+0.442639696 container cleanup a60500a596252b835a3f352020a68eee6d03bb718b960c3d3fbc3378dc09314c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:29:12.703 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 bash[46169]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b 2024-04-24T16:29:12.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:12 smithi156 ceph-mon[22679]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:12.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:12 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:12.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:12 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:12.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:12 smithi156 ceph-mon[22679]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-24T16:29:12.704 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:12 smithi156 ceph-mon[22679]: Reconfiguring daemon node-exporter.b on smithi156 2024-04-24T16:29:12.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[21827]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[21827]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[21827]: Reconfiguring daemon node-exporter.b on smithi156 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[26293]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[26293]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-24T16:29:12.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:12 smithi070 ceph-mon[26293]: Reconfiguring daemon node-exporter.b on smithi156 2024-04-24T16:29:12.965 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 podman[46183]: 2024-04-24 16:29:12.706770772 +0000 UTC m=+0.548717341 container remove a60500a596252b835a3f352020a68eee6d03bb718b960c3d3fbc3378dc09314c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:29:12.966 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:29:12.966 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 bash[46196]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-24T16:29:12.966 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-24T16:29:12.966 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 systemd[1]: Stopped Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:12.966 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 3.885s CPU time. 2024-04-24T16:29:12.966 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 systemd[1]: Starting Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:12 smithi156 podman[46270]: 2024-04-24 16:29:12.872855523 +0000 UTC m=+0.019161624 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 podman[46270]: 2024-04-24 16:29:13.002969988 +0000 UTC m=+0.149276073 container create cda127a7d141be473b3168aa67a355b4a02195e1554e550e904df266fc9936b8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 podman[46270]: 2024-04-24 16:29:13.184986355 +0000 UTC m=+0.331292440 container init cda127a7d141be473b3168aa67a355b4a02195e1554e550e904df266fc9936b8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 podman[46270]: 2024-04-24 16:29:13.187477099 +0000 UTC m=+0.333783184 container start cda127a7d141be473b3168aa67a355b4a02195e1554e550e904df266fc9936b8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.190Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.190Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-24T16:29:13.282 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-24T16:29:13.283 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=arp 2024-04-24T16:29:13.283 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-24T16:29:13.283 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-24T16:29:13.283 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-24T16:29:13.283 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-24T16:29:13.283 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-24T16:29:13.283 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=edac 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-24T16:29:13.284 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=os 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=stat 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-24T16:29:13.285 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=time 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=uname 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.191Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.192Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[46286]: ts=2024-04-24T16:29:13.192Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-24T16:29:13.286 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 bash[46270]: cda127a7d141be473b3168aa67a355b4a02195e1554e550e904df266fc9936b8 2024-04-24T16:29:13.540 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:29:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:29:13.111Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003467095s 2024-04-24T16:29:13.657 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:29:13 smithi156 systemd[1]: Started Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:14.495 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:14.495 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.495 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.495 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: Reconfiguring grafana.a (dependencies changed)... 2024-04-24T16:29:14.495 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-24T16:29:14.495 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.496 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.496 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:14 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:29:14.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: Reconfiguring grafana.a (dependencies changed)... 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:29:14.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:14.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: Reconfiguring grafana.a (dependencies changed)... 2024-04-24T16:29:14.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-24T16:29:14.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:14.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:14 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:29:15.380 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:15 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:29:15.380 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:15 smithi156 ceph-mon[22679]: Reconfiguring daemon grafana.a on smithi156 2024-04-24T16:29:15.380 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:15 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:15.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:15 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:29:15.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:15 smithi070 ceph-mon[26293]: Reconfiguring daemon grafana.a on smithi156 2024-04-24T16:29:15.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:15 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:15.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:15 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-24T16:29:15.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:15 smithi070 ceph-mon[21827]: Reconfiguring daemon grafana.a on smithi156 2024-04-24T16:29:15.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:15 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:16.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:16 smithi156 ceph-mon[22679]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:16.658 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:16 smithi156 systemd[1]: Stopping Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:16.658 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:16 smithi156 bash[46458]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-24T16:29:16.658 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[38842]: t=2024-04-24T16:29:16+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-24T16:29:16.658 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:16 smithi156 podman[46472]: 2024-04-24 16:29:16.526918778 +0000 UTC m=+0.145741948 container died 5a7c9a93bc9edf2b857d4b45cf4d91befaeb2859b99f85e41caeaed5de0c0090 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, io.buildah.version=1.24.2, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, description=Ceph Grafana Container, version=8.5, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.openshift.tags=base rhel8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-24T16:29:16.785 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:16 smithi070 ceph-mon[26293]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:16.786 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:16 smithi070 ceph-mon[21827]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:17.157 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:16 smithi156 podman[46472]: 2024-04-24 16:29:16.883304302 +0000 UTC m=+0.502127480 container cleanup 5a7c9a93bc9edf2b857d4b45cf4d91befaeb2859b99f85e41caeaed5de0c0090 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, version=8.5, distribution-scope=public, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-24T16:29:17.158 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:16 smithi156 bash[46472]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a 2024-04-24T16:29:17.158 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:16 smithi156 bash[46498]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-24T16:29:17.569 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 podman[46486]: 2024-04-24 16:29:17.192904266 +0000 UTC m=+0.665363768 container remove 5a7c9a93bc9edf2b857d4b45cf4d91befaeb2859b99f85e41caeaed5de0c0090 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, vcs-type=git, architecture=x86_64, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, name=ubi8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc.) 2024-04-24T16:29:17.569 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-24T16:29:17.570 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 systemd[1]: Stopped Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:17.570 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@grafana.a.service: Consumed 3.269s CPU time. 2024-04-24T16:29:17.570 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 systemd[1]: Starting Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:29:17.570 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 podman[46566]: 2024-04-24 16:29:17.327035705 +0000 UTC m=+0.017911509 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-24T16:29:17.570 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 podman[46566]: 2024-04-24 16:29:17.435873289 +0000 UTC m=+0.126749091 container create 3fc0342b47172515d6275e2fd0889184f5b8fe1b235e12b7217e6b31f0e7d062 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, version=8.5, release=236.1648460182, architecture=x86_64, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.openshift.expose-services=) 2024-04-24T16:29:17.985 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 podman[46566]: 2024-04-24 16:29:17.730353834 +0000 UTC m=+0.421229642 container init 3fc0342b47172515d6275e2fd0889184f5b8fe1b235e12b7217e6b31f0e7d062 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vendor=Red Hat, Inc., name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vcs-type=git, io.openshift.expose-services=) 2024-04-24T16:29:17.985 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 podman[46566]: 2024-04-24 16:29:17.733114836 +0000 UTC m=+0.423990639 container start 3fc0342b47172515d6275e2fd0889184f5b8fe1b235e12b7217e6b31f0e7d062 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, description=Ceph Grafana Container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, maintainer=Paul Cuzner , release=236.1648460182) 2024-04-24T16:29:17.985 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-24T16:29:17.985 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-24T16:29:17.985 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-24T16:29:17.985 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-24T16:29:17.985 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Bus STOPPING 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Bus STOPPED 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Bus STARTING 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Serving on http://:::9283 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Bus STARTED 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Bus STOPPING 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Bus STOPPED 2024-04-24T16:29:17.986 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:17] ENGINE Bus STARTING 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="App mode production" logger=settings 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-24T16:29:17.987 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=539.195µs 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 bash[46566]: 3fc0342b47172515d6275e2fd0889184f5b8fe1b235e12b7217e6b31f0e7d062 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 systemd[1]: Started Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-24T16:29:17.988 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-24T16:29:17.989 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:29:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:29:17+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-24T16:29:18.325 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE Serving on http://:::9283 2024-04-24T16:29:18.325 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE Bus STARTED 2024-04-24T16:29:18.325 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE Bus STOPPING 2024-04-24T16:29:18.325 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:29:18.325 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE Bus STOPPED 2024-04-24T16:29:18.325 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE Bus STARTING 2024-04-24T16:29:18.325 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE Serving on http://:::9283 2024-04-24T16:29:18.326 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:29:18] ENGINE Bus STARTED 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:29:18.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156:3000"}]: dispatch 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3318858542' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:29:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:29:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:29:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156:3000"}]: dispatch 2024-04-24T16:29:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:29:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:18 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3318858542' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:29:18.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:18.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156:3000"}]: dispatch 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:29:18.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:18 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3318858542' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:29:19.650 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:29:19.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-04-24T16:29:19.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:29:19.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:29:19.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:29:19.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156:3000"}]: dispatch 2024-04-24T16:29:19.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3000366481' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]: dispatch 2024-04-24T16:29:19.651 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:19 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]: dispatch 2024-04-24T16:29:19.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:29:19.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-04-24T16:29:19.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:29:19.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:29:19.714 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156:3000"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3000366481' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070.front.sepia.ceph.com:9093"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:29:19.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156:3000"}]: dispatch 2024-04-24T16:29:19.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3000366481' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]: dispatch 2024-04-24T16:29:19.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:19 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]: dispatch 2024-04-24T16:29:20.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[26293]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:29:20.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]': finished 2024-04-24T16:29:20.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[26293]: osdmap e96: 8 total, 8 up, 8 in 2024-04-24T16:29:20.540 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/731734138' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3930276067"}]: dispatch 2024-04-24T16:29:20.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[21827]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:29:20.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]': finished 2024-04-24T16:29:20.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[21827]: osdmap e96: 8 total, 8 up, 8 in 2024-04-24T16:29:20.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:20 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/731734138' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3930276067"}]: dispatch 2024-04-24T16:29:20.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:20 smithi156 ceph-mon[22679]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:29:20.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:20 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3427395814"}]': finished 2024-04-24T16:29:20.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:20 smithi156 ceph-mon[22679]: osdmap e96: 8 total, 8 up, 8 in 2024-04-24T16:29:20.657 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:20 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/731734138' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3930276067"}]: dispatch 2024-04-24T16:29:21.628 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:21 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/731734138' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3930276067"}]': finished 2024-04-24T16:29:21.628 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:21 smithi156 ceph-mon[22679]: osdmap e97: 8 total, 8 up, 8 in 2024-04-24T16:29:21.628 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:21 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/724139519' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]: dispatch 2024-04-24T16:29:21.628 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:21 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]: dispatch 2024-04-24T16:29:21.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/731734138' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3930276067"}]': finished 2024-04-24T16:29:21.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[21827]: osdmap e97: 8 total, 8 up, 8 in 2024-04-24T16:29:21.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/724139519' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]: dispatch 2024-04-24T16:29:21.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]: dispatch 2024-04-24T16:29:21.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/731734138' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3930276067"}]': finished 2024-04-24T16:29:21.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[26293]: osdmap e97: 8 total, 8 up, 8 in 2024-04-24T16:29:21.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/724139519' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]: dispatch 2024-04-24T16:29:21.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:21 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]: dispatch 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]': finished 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: osdmap e98: 8 total, 8 up, 8 in 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3652732381' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2241868720"}]: dispatch 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:29:22.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:22.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:22.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:22 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:22.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-24T16:29:22.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]': finished 2024-04-24T16:29:22.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: osdmap e98: 8 total, 8 up, 8 in 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3652732381' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2241868720"}]: dispatch 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:22.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/2453531589"}]': finished 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: osdmap e98: 8 total, 8 up, 8 in 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3652732381' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2241868720"}]: dispatch 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:29:22.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:22 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:29:23.729 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3652732381' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2241868720"}]': finished 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[26293]: osdmap e99: 8 total, 8 up, 8 in 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2136282728' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2234875754"}]: dispatch 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3652732381' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2241868720"}]': finished 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[21827]: osdmap e99: 8 total, 8 up, 8 in 2024-04-24T16:29:23.730 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:23 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2136282728' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2234875754"}]: dispatch 2024-04-24T16:29:23.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:29:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:23 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:29:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:23 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:29:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:23 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3652732381' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2241868720"}]': finished 2024-04-24T16:29:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:23 smithi156 ceph-mon[22679]: osdmap e99: 8 total, 8 up, 8 in 2024-04-24T16:29:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:23 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2136282728' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2234875754"}]: dispatch 2024-04-24T16:29:24.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:24 smithi070 ceph-mon[26293]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:29:24.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:24 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2136282728' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2234875754"}]': finished 2024-04-24T16:29:24.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:24 smithi070 ceph-mon[26293]: osdmap e100: 8 total, 8 up, 8 in 2024-04-24T16:29:24.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:24 smithi070 ceph-mon[21827]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:29:24.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:24 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2136282728' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2234875754"}]': finished 2024-04-24T16:29:24.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:24 smithi070 ceph-mon[21827]: osdmap e100: 8 total, 8 up, 8 in 2024-04-24T16:29:24.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:24 smithi156 ceph-mon[22679]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:29:24.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:24 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2136282728' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2234875754"}]': finished 2024-04-24T16:29:24.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:24 smithi156 ceph-mon[22679]: osdmap e100: 8 total, 8 up, 8 in 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:29:25.373+0000 7f85a1eb6700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:25.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:25.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:25.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:25.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:25.662 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:25.663 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:29:26] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-04-24T16:29:26.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:26 smithi156 ceph-mon[22679]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-04-24T16:29:26.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:26 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:26.658 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:26.658 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:26.658 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:26.658 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:26.658 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:26.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:29:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:26.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:26 smithi070 ceph-mon[26293]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:26 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:26.791 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:26.792 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:26.793 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.795 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:26 smithi070 ceph-mon[21827]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:26 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:26.796 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.797 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:29:26.798 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:29:26.799 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:29:26.800 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:29:28.790 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:28 smithi070 ceph-mon[21827]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:28.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:28 smithi070 ceph-mon[26293]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:28.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:28 smithi156 ceph-mon[22679]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:28.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:29:30.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:30 smithi070 ceph-mon[26293]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-04-24T16:29:30.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:30 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:30 smithi070 ceph-mon[21827]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-04-24T16:29:30.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:30 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:30.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:30 smithi156 ceph-mon[22679]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-04-24T16:29:30.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:30 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:32.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:32 smithi070 ceph-mon[26293]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-24T16:29:32.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:32 smithi070 ceph-mon[21827]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-24T16:29:32.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:32 smithi156 ceph-mon[22679]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-24T16:29:33.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:29:33.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:33 smithi156 ceph-mon[22679]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:29:34.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:33 smithi070 ceph-mon[21827]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:29:34.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:33 smithi070 ceph-mon[26293]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:29:36.111 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:35 smithi070 ceph-mon[26293]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 463 B/s rd, 0 op/s 2024-04-24T16:29:36.111 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:35 smithi070 ceph-mon[21827]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 463 B/s rd, 0 op/s 2024-04-24T16:29:36.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:35 smithi156 ceph-mon[22679]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 463 B/s rd, 0 op/s 2024-04-24T16:29:36.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:29:36] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-04-24T16:29:38.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:37 smithi156 ceph-mon[22679]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:37 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3049711177' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:29:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:37 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3929609487' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]: dispatch 2024-04-24T16:29:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:37 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]: dispatch 2024-04-24T16:29:38.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[21827]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:38.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3049711177' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:29:38.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3929609487' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]: dispatch 2024-04-24T16:29:38.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]: dispatch 2024-04-24T16:29:38.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[26293]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:29:38.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3049711177' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:29:38.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3929609487' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]: dispatch 2024-04-24T16:29:38.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:37 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]: dispatch 2024-04-24T16:29:38.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:38 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]': finished 2024-04-24T16:29:38.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:38 smithi156 ceph-mon[22679]: osdmap e101: 8 total, 8 up, 8 in 2024-04-24T16:29:38.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:38 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4132092238' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2271065937"}]: dispatch 2024-04-24T16:29:38.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:29:39.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:38 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]': finished 2024-04-24T16:29:39.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:38 smithi070 ceph-mon[26293]: osdmap e101: 8 total, 8 up, 8 in 2024-04-24T16:29:39.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:38 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4132092238' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2271065937"}]: dispatch 2024-04-24T16:29:39.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:38 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/2453531589"}]': finished 2024-04-24T16:29:39.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:38 smithi070 ceph-mon[21827]: osdmap e101: 8 total, 8 up, 8 in 2024-04-24T16:29:39.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:38 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4132092238' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2271065937"}]: dispatch 2024-04-24T16:29:40.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:39 smithi156 ceph-mon[22679]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:29:40.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:39 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4132092238' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2271065937"}]': finished 2024-04-24T16:29:40.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:39 smithi156 ceph-mon[22679]: osdmap e102: 8 total, 8 up, 8 in 2024-04-24T16:29:40.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:39 smithi070 ceph-mon[21827]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:29:40.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:39 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4132092238' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2271065937"}]': finished 2024-04-24T16:29:40.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:39 smithi070 ceph-mon[21827]: osdmap e102: 8 total, 8 up, 8 in 2024-04-24T16:29:40.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:39 smithi070 ceph-mon[26293]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:29:40.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:39 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4132092238' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2271065937"}]': finished 2024-04-24T16:29:40.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:39 smithi070 ceph-mon[26293]: osdmap e102: 8 total, 8 up, 8 in 2024-04-24T16:29:42.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:41 smithi070 ceph-mon[26293]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:42.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:41 smithi070 ceph-mon[21827]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:42.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:41 smithi156 ceph-mon[22679]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:43.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:43 smithi156 ceph-mon[22679]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:43.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:43 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:29:44.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:43 smithi070 ceph-mon[26293]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:44.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:43 smithi070 ceph-mon[21827]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:46.062 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:45 smithi156 ceph-mon[22679]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:46.063 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:46.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:45 smithi070 ceph-mon[26293]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:46.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:46.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:45 smithi070 ceph-mon[21827]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:29:46.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:29:46.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:29:46] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-24T16:29:48.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:47 smithi156 ceph-mon[22679]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:48.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:47 smithi156 ceph-mon[22679]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:29:48.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:47 smithi070 ceph-mon[26293]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:48.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:47 smithi070 ceph-mon[26293]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:29:48.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:47 smithi070 ceph-mon[21827]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:48.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:47 smithi070 ceph-mon[21827]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:29:48.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:29:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:49 smithi156 ceph-mon[22679]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-24T16:29:50.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:49 smithi070 ceph-mon[26293]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-24T16:29:50.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:49 smithi070 ceph-mon[21827]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-24T16:29:52.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:51 smithi156 ceph-mon[22679]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:29:52.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:51 smithi070 ceph-mon[21827]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:29:52.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:51 smithi070 ceph-mon[26293]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:29:53.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:53 smithi156 ceph-mon[22679]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:53.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:29:54.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:53 smithi070 ceph-mon[21827]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:54.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:53 smithi070 ceph-mon[26293]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:56.131 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:55 smithi156 ceph-mon[22679]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:56.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:55 smithi070 ceph-mon[21827]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:56.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:55 smithi070 ceph-mon[26293]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:29:56.407 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:29:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:29:56] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-24T16:29:58.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:57 smithi156 ceph-mon[22679]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:58.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:57 smithi156 ceph-mon[22679]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:29:58.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:57 smithi070 ceph-mon[21827]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:58.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:57 smithi070 ceph-mon[21827]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:29:58.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:57 smithi070 ceph-mon[26293]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:29:58.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:57 smithi070 ceph-mon[26293]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:29:58.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:29:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:29:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:00.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:29:59 smithi156 ceph-mon[22679]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:00.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:29:59 smithi070 ceph-mon[26293]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:00.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:29:59 smithi070 ceph-mon[21827]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:01.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:00 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:01.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:00 smithi156 ceph-mon[22679]: overall HEALTH_OK 2024-04-24T16:30:01.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:00 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:01.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:00 smithi070 ceph-mon[26293]: overall HEALTH_OK 2024-04-24T16:30:01.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:00 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:01.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:00 smithi070 ceph-mon[21827]: overall HEALTH_OK 2024-04-24T16:30:02.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:01 smithi156 ceph-mon[22679]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:02.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:01 smithi070 ceph-mon[26293]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:02.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:01 smithi070 ceph-mon[21827]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:03.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:03 smithi156 ceph-mon[22679]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:03.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:04.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:03 smithi070 ceph-mon[26293]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:03 smithi070 ceph-mon[21827]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:06.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:05 smithi156 ceph-mon[22679]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:06.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:05 smithi070 ceph-mon[26293]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:06.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:05 smithi070 ceph-mon[21827]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:06.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:30:06] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-04-24T16:30:08.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:07 smithi156 ceph-mon[22679]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:08.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:07 smithi156 ceph-mon[22679]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:08.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:07 smithi070 ceph-mon[26293]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:08.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:07 smithi070 ceph-mon[26293]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:08.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:07 smithi070 ceph-mon[21827]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:08.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:07 smithi070 ceph-mon[21827]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:08.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:10.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:09 smithi156 ceph-mon[22679]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:10.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:09 smithi070 ceph-mon[26293]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:10.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:09 smithi070 ceph-mon[21827]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:12.092 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:11 smithi070 ceph-mon[26293]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:12.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:11 smithi070 ceph-mon[21827]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:12.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:11 smithi156 ceph-mon[22679]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:13.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:13 smithi156 ceph-mon[22679]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:13.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:14.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:13 smithi070 ceph-mon[26293]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:14.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:13 smithi070 ceph-mon[21827]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:15 smithi156 ceph-mon[22679]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:16.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:15 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:16.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:15 smithi070 ceph-mon[26293]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:16.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:15 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:16.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:15 smithi070 ceph-mon[21827]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:30:16.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:15 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:16.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:30:16] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-04-24T16:30:18.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:17 smithi156 ceph-mon[22679]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:17 smithi156 ceph-mon[22679]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:18.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:17 smithi070 ceph-mon[26293]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:18.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:17 smithi070 ceph-mon[26293]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:18.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:17 smithi070 ceph-mon[21827]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:18.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:17 smithi070 ceph-mon[21827]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:18.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:20.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:19 smithi156 ceph-mon[22679]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:20.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:19 smithi070 ceph-mon[26293]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:19 smithi070 ceph-mon[21827]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:22.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:21 smithi156 ceph-mon[22679]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:22.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:21 smithi070 ceph-mon[26293]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:22.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:21 smithi070 ceph-mon[21827]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:23 smithi156 ceph-mon[22679]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:23.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:24.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:23 smithi070 ceph-mon[21827]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:24.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:23 smithi070 ceph-mon[26293]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:26.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:25 smithi156 ceph-mon[22679]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:26.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:25 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:30:26.163 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:25 smithi070 ceph-mon[21827]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:26.163 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:25 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:30:26.163 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:25 smithi070 ceph-mon[26293]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:26.163 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:25 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:30:26.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:30:26] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-04-24T16:30:27.907 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:27 smithi156 ceph-mon[22679]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:27.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:27 smithi156 ceph-mon[22679]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:28.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:27 smithi070 ceph-mon[26293]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:28.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:27 smithi070 ceph-mon[26293]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:28.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:27 smithi070 ceph-mon[21827]: from='client.25027 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:28.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:27 smithi070 ceph-mon[21827]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:28.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:30.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[21827]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:30.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[26293]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:30 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:30.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:30 smithi156 ceph-mon[22679]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:30.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:30 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:30 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:30 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:30 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:30.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:30 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:30:31.084 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:30:31.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:31 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:30:31.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:31 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:30:32.407 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:32 smithi156 ceph-mon[22679]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:32.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:32 smithi070 ceph-mon[26293]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:32.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:32 smithi070 ceph-mon[21827]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:30:33.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:33.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-mon[22679]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:30:33.909 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:30:33.910 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:33.911 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:33.912 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:33.913 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:33.913 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:33.913 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:33.913 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:33.913 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.913 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:33.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:30:33.513+0000 7f85a1eb6700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:30:33.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:30:33.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:30:33.916 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:30:33.917 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:33.918 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:33 smithi070 ceph-mon[26293]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:33 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:30:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:34.042 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:30:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:34.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:34.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:33 smithi070 ceph-mon[21827]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:33 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:34.046 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:34.047 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:30:34.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:34.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:30:34.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:30:36.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:35 smithi156 ceph-mon[22679]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:36.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:35 smithi070 ceph-mon[26293]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:36.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:35 smithi070 ceph-mon[21827]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:30:36.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:30:36] "GET /metrics HTTP/1.1" 200 37887 "" "Prometheus/2.33.4" 2024-04-24T16:30:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:37 smithi156 ceph-mon[22679]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:30:38.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:37 smithi070 ceph-mon[21827]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:30:38.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:37 smithi070 ceph-mon[26293]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:30:38.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:40.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:39 smithi156 ceph-mon[22679]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:30:40.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:39 smithi070 ceph-mon[21827]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:30:40.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:39 smithi070 ceph-mon[26293]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:30:42.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:41 smithi070 ceph-mon[21827]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:30:42.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:41 smithi070 ceph-mon[26293]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:30:42.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:41 smithi156 ceph-mon[22679]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:30:43.822 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:43 smithi070 ceph-mon[21827]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-24T16:30:43.822 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:43 smithi070 ceph-mon[26293]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-24T16:30:43.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:43 smithi156 ceph-mon[22679]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-24T16:30:43.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:43 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:45 smithi156 ceph-mon[22679]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-24T16:30:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:45 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:45 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3692753270' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:30:46.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:45 smithi070 ceph-mon[26293]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-24T16:30:46.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:45 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:46.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:45 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3692753270' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:30:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:45 smithi070 ceph-mon[21827]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-24T16:30:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:45 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:30:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:45 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3692753270' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:30:46.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:30:46] "GET /metrics HTTP/1.1" 200 37886 "" "Prometheus/2.33.4" 2024-04-24T16:30:48.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:47 smithi156 ceph-mon[22679]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:30:48.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:47 smithi070 ceph-mon[26293]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:30:48.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:47 smithi070 ceph-mon[21827]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:30:48.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:50.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:49 smithi156 ceph-mon[22679]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-24T16:30:50.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:49 smithi070 ceph-mon[21827]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-24T16:30:50.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:49 smithi070 ceph-mon[26293]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-24T16:30:52.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:51 smithi156 ceph-mon[22679]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:52.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:51 smithi070 ceph-mon[26293]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:52.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:51 smithi070 ceph-mon[21827]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:53.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:53 smithi156 ceph-mon[22679]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:53.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:30:54.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:53 smithi070 ceph-mon[26293]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:53 smithi070 ceph-mon[21827]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:56.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:55 smithi156 ceph-mon[22679]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:56.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:55 smithi156 ceph-mon[22679]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:56.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:55 smithi070 ceph-mon[26293]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:56.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:55 smithi070 ceph-mon[26293]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:56.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:55 smithi070 ceph-mon[21827]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:30:56.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:55 smithi070 ceph-mon[21827]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:30:56.657 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:30:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:30:56] "GET /metrics HTTP/1.1" 200 37886 "" "Prometheus/2.33.4" 2024-04-24T16:30:58.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:57 smithi156 ceph-mon[22679]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:30:58.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:57 smithi070 ceph-mon[26293]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:30:58.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:57 smithi070 ceph-mon[21827]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:30:58.907 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:30:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:30:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:30:59 smithi156 ceph-mon[22679]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:00.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:30:59 smithi070 ceph-mon[26293]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:00.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:30:59 smithi070 ceph-mon[21827]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:01.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:00 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:01.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:00 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:01.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:00 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:02.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:01 smithi156 ceph-mon[22679]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:02.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:01 smithi070 ceph-mon[26293]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:02.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:01 smithi070 ceph-mon[21827]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:03.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:03 smithi156 ceph-mon[22679]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:03.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:04.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:03 smithi070 ceph-mon[26293]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:03 smithi070 ceph-mon[21827]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:06.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:05 smithi156 ceph-mon[22679]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:06.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:05 smithi156 ceph-mon[22679]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:06.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:05 smithi070 ceph-mon[21827]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:06.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:05 smithi070 ceph-mon[21827]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:06.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:05 smithi070 ceph-mon[26293]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:06.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:05 smithi070 ceph-mon[26293]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:06.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:31:06] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-04-24T16:31:08.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:07 smithi156 ceph-mon[22679]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:08.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:07 smithi070 ceph-mon[26293]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:08.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:07 smithi070 ceph-mon[21827]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:08.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:10.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:09 smithi156 ceph-mon[22679]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:10.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:09 smithi070 ceph-mon[26293]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:09 smithi070 ceph-mon[21827]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:12.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:11 smithi156 ceph-mon[22679]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:12.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:11 smithi070 ceph-mon[26293]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:12.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:11 smithi070 ceph-mon[21827]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:13.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:13 smithi156 ceph-mon[22679]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:13.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:14.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:13 smithi070 ceph-mon[21827]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:14.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:13 smithi070 ceph-mon[26293]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:15 smithi156 ceph-mon[22679]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:15 smithi156 ceph-mon[22679]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:15 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:16.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:15 smithi070 ceph-mon[26293]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:16.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:15 smithi070 ceph-mon[26293]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:16.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:15 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:16.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:15 smithi070 ceph-mon[21827]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:16.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:15 smithi070 ceph-mon[21827]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:16.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:15 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:16.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:31:16] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-24T16:31:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:17 smithi156 ceph-mon[22679]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:18.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:17 smithi070 ceph-mon[21827]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:18.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:17 smithi070 ceph-mon[26293]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:18.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:20.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:19 smithi156 ceph-mon[22679]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:20.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:19 smithi070 ceph-mon[21827]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:20.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:19 smithi070 ceph-mon[26293]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:22.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:21 smithi156 ceph-mon[22679]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:22.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:21 smithi070 ceph-mon[21827]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:22.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:21 smithi070 ceph-mon[26293]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:23.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:23 smithi156 ceph-mon[22679]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:24.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:23 smithi070 ceph-mon[21827]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:23 smithi070 ceph-mon[26293]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:25 smithi156 ceph-mon[22679]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:25 smithi156 ceph-mon[22679]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:26.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:25 smithi070 ceph-mon[21827]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:26.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:25 smithi070 ceph-mon[21827]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:26.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:25 smithi070 ceph-mon[26293]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:26.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:25 smithi070 ceph-mon[26293]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:26.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:31:26] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-24T16:31:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:27 smithi156 ceph-mon[22679]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:28.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:27 smithi070 ceph-mon[21827]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:28.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:27 smithi070 ceph-mon[26293]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:28.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:29 smithi156 ceph-mon[22679]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:29 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:29 smithi070 ceph-mon[26293]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:29 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:29 smithi070 ceph-mon[21827]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:29 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:31 smithi156 ceph-mon[22679]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:32.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:31 smithi070 ceph-mon[26293]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:31 smithi070 ceph-mon[21827]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:33.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:33 smithi156 ceph-mon[22679]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:33.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:33 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:31:33.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:34.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:33 smithi070 ceph-mon[26293]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:33 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:31:34.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:33 smithi070 ceph-mon[21827]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:34.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:33 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:31:36.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:35 smithi156 ceph-mon[22679]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:36.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:35 smithi156 ceph-mon[22679]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:36.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:35 smithi070 ceph-mon[21827]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:36.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:35 smithi070 ceph-mon[21827]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:36.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:35 smithi070 ceph-mon[26293]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:36.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:35 smithi070 ceph-mon[26293]: from='client.34835 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:31:36.640 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:31:36] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-04-24T16:31:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:37 smithi156 ceph-mon[22679]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:37 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:37 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:38.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:37 smithi070 ceph-mon[26293]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:38.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:37 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:38.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:37 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:38.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:37 smithi070 ceph-mon[21827]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:31:38.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:37 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:38.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:37 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:38.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:38.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:31:38.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:31:38.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:38.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:31:38.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:31:38.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:31:38.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:31:38.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:38 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:31:39.170 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:31:39.170 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:31:39.170 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:31:39.171 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:31:39.172 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:31:39.172 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:38 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:31:40.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:40 smithi070 ceph-mon[26293]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:40.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:40 smithi070 ceph-mon[21827]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:40.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:40 smithi156 ceph-mon[22679]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:31:41.908 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:31:41.508+0000 7f85a1eb6700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:31:41.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:41.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:41.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:41.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:41.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:41.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:41.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:31:41.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:41.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:41.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:41.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:41.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:41.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:41.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:41 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:42 smithi070 ceph-mon[26293]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:42 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.541 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:42.542 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:31:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.545 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:42 smithi070 ceph-mon[21827]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:42 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:31:42.546 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:31:42.547 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:31:42.548 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:42.549 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:42.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:42.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:42.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:42.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:42.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:42.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.550 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:42 smithi156 ceph-mon[22679]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:42 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.660 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:31:42.661 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:31:42.662 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:31:42.663 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:31:43.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:43 smithi156 ceph-mon[22679]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:43.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:43 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:44.040 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:43 smithi070 ceph-mon[21827]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:43 smithi070 ceph-mon[26293]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:45.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:44 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:45.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:44 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:44 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:31:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:45 smithi156 ceph-mon[22679]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:46.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:45 smithi070 ceph-mon[26293]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:45 smithi070 ceph-mon[21827]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:46.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:31:46] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-04-24T16:31:47.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-24T16:31:48.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:47 smithi156 ceph-mon[22679]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:48.223 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:47 smithi070 ceph-mon[26293]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:48.223 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:47 smithi070 ceph-mon[21827]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-24T16:31:48.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:49.892 INFO:teuthology.orchestra.run.smithi070.stdout:Scheduled to redeploy mgr.y on host 'smithi070' 2024-04-24T16:31:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:49 smithi156 ceph-mon[22679]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:31:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:49 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/734284344' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:31:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:49 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:49 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:49 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:49 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[26293]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/734284344' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[21827]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/734284344' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T16:31:50.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:50.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:49 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:31:50.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps --refresh' 2024-04-24T16:31:51.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:50 smithi070 ceph-mon[26293]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:31:51.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:50 smithi070 ceph-mon[26293]: Schedule redeploy daemon mgr.y 2024-04-24T16:31:51.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:50 smithi070 ceph-mon[21827]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:31:51.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:50 smithi070 ceph-mon[21827]: Schedule redeploy daemon mgr.y 2024-04-24T16:31:51.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:50 smithi156 ceph-mon[22679]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:31:51.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:50 smithi156 ceph-mon[22679]: Schedule redeploy daemon mgr.y 2024-04-24T16:31:52.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:51 smithi156 ceph-mon[22679]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:31:52.169 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:51 smithi070 ceph-mon[26293]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:31:52.170 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:51 smithi070 ceph-mon[21827]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-24T16:31:52.607 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:31:52.607 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (2m) 83s ago 10m 13.9M - 0.23.0 ba2b418f427c 609b591dc19c 2024-04-24T16:31:52.607 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (2m) 14s ago 10m 47.3M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (2m) 83s ago 9m 46.1M - 3.5 e1d6a67b021e 7585d4bbb08b 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283 running (5m) 14s ago 14m 510M - 19.0.0-3285-g5904d294 8bb2087275ff 432b981bb958 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:9283 running (16m) 83s ago 16m 410M - 17.2.0 e1d6a67b021e cbef4307b27d 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (16m) 83s ago 16m 53.1M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (15m) 14s ago 15m 49.3M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (15m) 83s ago 15m 44.4M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (2m) 83s ago 11m 14.2M - 1.3.1 1dbe0e931976 a0dadc7082e4 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (2m) 14s ago 11m 15.9M - 1.3.1 1dbe0e931976 cda127a7d141 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (14m) 83s ago 14m 54.1M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (14m) 83s ago 14m 55.1M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (13m) 83s ago 13m 49.2M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (13m) 83s ago 13m 49.7M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (13m) 14s ago 13m 50.7M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:31:52.608 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (12m) 14s ago 12m 50.3M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:31:52.609 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (12m) 14s ago 12m 47.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:31:52.609 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (11m) 14s ago 11m 52.1M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:31:52.609 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (2m) 14s ago 11m 70.4M - 2.33.4 514e6a882f6e 0e9a48e5d60e 2024-04-24T16:31:52.609 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (9m) 83s ago 9m 71.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:31:52.609 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (9m) 14s ago 9m 71.8M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:31:53.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:53.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:53 smithi156 ceph-mon[22679]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:31:53.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:53 smithi156 ceph-mon[22679]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:53.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:53 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:54.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:53 smithi070 ceph-mon[26293]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:31:54.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:53 smithi070 ceph-mon[26293]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:54.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:53 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:53 smithi070 ceph-mon[21827]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:31:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:53 smithi070 ceph-mon[21827]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:53 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:54.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'sleep 180' 2024-04-24T16:31:55.933 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:55 smithi070 ceph-mon[26293]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:55.933 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:55 smithi070 ceph-mon[21827]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:56.070 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:55 smithi156 ceph-mon[22679]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:56.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:31:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:31:56] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-04-24T16:31:56.949 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:56 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3772762462' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:31:57.040 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:56 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3772762462' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:31:57.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:56 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3772762462' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:31:58.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:57 smithi070 ceph-mon[21827]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:58.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:57 smithi070 ceph-mon[26293]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:58.157 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:57 smithi156 ceph-mon[22679]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:31:58.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:58 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:58.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:58 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:58.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:31:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:31:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:31:59.276 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:58 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:59.276 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:58 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:59.276 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:58 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:31:59.276 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:58 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:32:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:00.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:32:00.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:32:00.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:32:00.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:32:00.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:00.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:31:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:00.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:32:00.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:00.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:32:00.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:32:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:32:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:32:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:31:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:01.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:00 smithi070 ceph-mon[21827]: Deploying daemon mgr.y on smithi070 2024-04-24T16:32:01.242 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:00 smithi070 ceph-mon[26293]: Deploying daemon mgr.y on smithi070 2024-04-24T16:32:01.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:00 smithi156 ceph-mon[22679]: Deploying daemon mgr.y on smithi070 2024-04-24T16:32:01.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:01 smithi070 systemd[1]: Stopping Ceph mgr.y for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:32:02.096 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:01 smithi070 ceph-mon[26293]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:02.097 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:01 smithi070 ceph-mon[21827]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:02.097 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:01 smithi070 podman[60410]: 2024-04-24 16:32:01.794163619 +0000 UTC m=+0.120091150 container died cbef4307b27d44b46080bf153021809553300fda247b84ff56d830405ee0bc5e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, com.redhat.component=centos-stream-container, vcs-type=git, version=8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_BRANCH=HEAD, RELEASE=HEAD, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-24T16:32:02.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:01 smithi156 ceph-mon[22679]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:02.439 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 podman[60410]: 2024-04-24 16:32:02.173673787 +0000 UTC m=+0.499601314 container cleanup cbef4307b27d44b46080bf153021809553300fda247b84ff56d830405ee0bc5e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, ceph=True, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , name=centos-stream, GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, distribution-scope=public, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8) 2024-04-24T16:32:02.440 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 bash[60410]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y 2024-04-24T16:32:02.746 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 podman[60424]: 2024-04-24 16:32:02.442389943 +0000 UTC m=+0.640470693 container remove cbef4307b27d44b46080bf153021809553300fda247b84ff56d830405ee0bc5e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, io.openshift.expose-services=, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, ceph=True, release=754, distribution-scope=public, vendor=Red Hat, Inc., RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, version=8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8) 2024-04-24T16:32:02.746 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:32:02.746 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-24T16:32:02.746 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 systemd[1]: Stopped Ceph mgr.y for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:32:02.746 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service: Consumed 59.155s CPU time. 2024-04-24T16:32:03.007 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 systemd[1]: Starting Ceph mgr.y for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:32:03.007 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:02 smithi070 podman[60558]: 2024-04-24 16:32:02.867696177 +0000 UTC m=+0.017862507 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:32:03.277 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 podman[60558]: 2024-04-24 16:32:03.01018007 +0000 UTC m=+0.160346391 container create e33f2bccae2e2df01228af6e8d2a17bf96bb5c609e7f882f34d09716568bc6a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-24T16:32:03.277 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 podman[60558]: 2024-04-24 16:32:03.279846036 +0000 UTC m=+0.430012354 container init e33f2bccae2e2df01228af6e8d2a17bf96bb5c609e7f882f34d09716568bc6a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-24T16:32:03.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 podman[60558]: 2024-04-24 16:32:03.282501366 +0000 UTC m=+0.432667703 container start e33f2bccae2e2df01228af6e8d2a17bf96bb5c609e7f882f34d09716568bc6a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=) 2024-04-24T16:32:03.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 ceph-mgr[60576]: -- 172.21.15.70:0/2284060831 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eba25c85a0 con 0x55eba259cc00 2024-04-24T16:32:03.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 bash[60558]: e33f2bccae2e2df01228af6e8d2a17bf96bb5c609e7f882f34d09716568bc6a0 2024-04-24T16:32:03.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 systemd[1]: Started Ceph mgr.y for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:32:03.541 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:03.444+0000 7f8c91703200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:32:03.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:04.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:03.640+0000 7f8c91703200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:32:04.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:03.749+0000 7f8c91703200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:32:04.338 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:04.079+0000 7f8c91703200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:32:04.338 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:04.167+0000 7f8c91703200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:04.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:04.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:04 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:32:04.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:04.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:04 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:32:05.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:05 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:05.455+0000 7f8c91703200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:32:05.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:05 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:05.628+0000 7f8c91703200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:32:06.108 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:05 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:05.800+0000 7f8c91703200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:32:06.411 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:06.110+0000 7f8c91703200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:32:06.411 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:06.206+0000 7f8c91703200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:32:06.411 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:06.413+0000 7f8c91703200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:32:06.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:06 smithi156 ceph-mon[22679]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:06.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:32:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:32:06] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-04-24T16:32:06.790 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:06 smithi070 ceph-mon[26293]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:06.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:06 smithi070 ceph-mon[21827]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:32:06.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:06.612+0000 7f8c91703200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:32:07.383 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:07 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:07.260+0000 7f8c91703200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:32:08.040 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:07 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:07.716+0000 7f8c91703200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:32:08.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:07 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:07.816+0000 7f8c91703200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:32:08.438 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:08.051+0000 7f8c91703200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:32:08.724 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:08.623+0000 7f8c91703200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:32:08.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[26293]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:08.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:08.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:08.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:32:08.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:32:08.724 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:32:08.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[21827]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:08.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:08.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:08.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:32:08.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:32:08.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:32:08.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:08.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:08 smithi156 ceph-mon[22679]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:08.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:08 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:08.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:08 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:08.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:08 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:32:08.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:08 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:32:08.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:08 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:32:08.976 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:08.725+0000 7f8c91703200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:32:08.976 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:08 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:08.977+0000 7f8c91703200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:32:09.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:09.119+0000 7f8c91703200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:32:09.291 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:09.218+0000 7f8c91703200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:32:09.653 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:32:09.653 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:32:09.653 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2211661587' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:32:09.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:32:09.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:32:09.653 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2211661587' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:32:09.654 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:09.340+0000 7f8c91703200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:32:09.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:09 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:32:09.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:09 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:32:09.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:09 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2211661587' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:32:10.041 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:09 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:09.656+0000 7f8c91703200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:32:10.581 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-mon[26293]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:10.581 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:10.306+0000 7f8c91703200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:32:10.581 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:10.551+0000 7f8c91703200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:32:10.582 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:32:10] ENGINE Bus STARTING 2024-04-24T16:32:10.582 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-mon[21827]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:10.840 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: CherryPy Checker: 2024-04-24T16:32:10.840 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: The Application mounted at '' has an empty config. 2024-04-24T16:32:10.841 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:32:10.841 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:32:10] ENGINE Serving on http://:::9283 2024-04-24T16:32:10.841 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:32:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:32:10] ENGINE Bus STARTED 2024-04-24T16:32:10.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:10 smithi156 ceph-mon[22679]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[21827]: Standby manager daemon y restarted 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[21827]: Standby manager daemon y started 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[26293]: Standby manager daemon y restarted 2024-04-24T16:32:11.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[26293]: Standby manager daemon y started 2024-04-24T16:32:11.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:32:11.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:32:11.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:32:11.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:11 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:32:11.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:11 smithi156 ceph-mon[22679]: Standby manager daemon y restarted 2024-04-24T16:32:11.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:11 smithi156 ceph-mon[22679]: Standby manager daemon y started 2024-04-24T16:32:11.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:11 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:32:11.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:11 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:32:11.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:11 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:32:11.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:11 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:32:12.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:12 smithi070 ceph-mon[26293]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:12.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:12 smithi070 ceph-mon[26293]: mgrmap e27: x(active, since 3m), standbys: y 2024-04-24T16:32:12.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:12 smithi070 ceph-mon[21827]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:12.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:12 smithi070 ceph-mon[21827]: mgrmap e27: x(active, since 3m), standbys: y 2024-04-24T16:32:12.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:12 smithi156 ceph-mon[22679]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:12.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:12 smithi156 ceph-mon[22679]: mgrmap e27: x(active, since 3m), standbys: y 2024-04-24T16:32:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:32:13.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:13.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:32:13.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:13 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.810 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:13.810 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:13 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.810 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:13 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:13.810 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:13 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:32:13.810 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:13 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:32:13.810 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:13 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:32:14.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:14 smithi070 ceph-mon[26293]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:32:14.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:14 smithi070 ceph-mon[21827]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:32:14.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:14 smithi156 ceph-mon[22679]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:32:15.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:15 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:15.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:15 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:15.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:15 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:16.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:16 smithi156 ceph-mon[22679]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:32:16.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:32:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:32:16] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-04-24T16:32:16.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:16 smithi070 ceph-mon[26293]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:32:16.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:16 smithi070 ceph-mon[21827]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:32:18.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:18 smithi070 ceph-mon[26293]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:18.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:18 smithi070 ceph-mon[21827]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:18.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:18 smithi156 ceph-mon[22679]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:32:18.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:19.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:19 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:19.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:19 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:19.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:19 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:20.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:20 smithi070 ceph-mon[26293]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:32:20.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:20 smithi070 ceph-mon[21827]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:32:20.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:20 smithi156 ceph-mon[22679]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-24T16:32:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:22 smithi070 ceph-mon[26293]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:32:22.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:22 smithi070 ceph-mon[21827]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:32:22.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:22 smithi156 ceph-mon[22679]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:32:23.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:23 smithi156 ceph-mon[22679]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:23.909 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:23 smithi070 ceph-mon[21827]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:23 smithi070 ceph-mon[26293]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:25 smithi156 ceph-mon[22679]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:26.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:25 smithi070 ceph-mon[21827]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:26.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:25 smithi070 ceph-mon[26293]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:26.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:32:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:32:26] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-04-24T16:32:28.128 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:27 smithi070 ceph-mon[26293]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:28.128 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:27 smithi070 ceph-mon[21827]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:27 smithi156 ceph-mon[22679]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:28.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:28 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:28.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:29.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:28 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:28 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:29 smithi156 ceph-mon[22679]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:29 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:29 smithi070 ceph-mon[21827]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:29 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:29 smithi070 ceph-mon[26293]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:29 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:31 smithi156 ceph-mon[22679]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:31 smithi070 ceph-mon[21827]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:32.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:31 smithi070 ceph-mon[26293]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:33.756 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:34.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:33 smithi070 ceph-mon[21827]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:33 smithi070 ceph-mon[26293]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:34.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:33 smithi156 ceph-mon[22679]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:36.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:35 smithi156 ceph-mon[22679]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:36.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:35 smithi070 ceph-mon[21827]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:36.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:35 smithi070 ceph-mon[26293]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:36.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:32:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:32:36] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-04-24T16:32:38.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:37 smithi070 ceph-mon[26293]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:38.139 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:37 smithi070 ceph-mon[21827]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:37 smithi156 ceph-mon[22679]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:38.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:38 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:38.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:39.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:38 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:39.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:38 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:40.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:39 smithi156 ceph-mon[22679]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:40.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:39 smithi070 ceph-mon[26293]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:40.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:39 smithi070 ceph-mon[21827]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:42.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:41 smithi156 ceph-mon[22679]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:42.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:41 smithi070 ceph-mon[26293]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:42.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:41 smithi070 ceph-mon[21827]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:43.756 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:43 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:43 smithi070 ceph-mon[21827]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:43 smithi070 ceph-mon[26293]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:44.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:43 smithi156 ceph-mon[22679]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:45.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:44 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:45.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:44 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:44 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:32:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:45 smithi156 ceph-mon[22679]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:46.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:45 smithi070 ceph-mon[26293]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:45 smithi070 ceph-mon[21827]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:46.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:32:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:32:46] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-04-24T16:32:48.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:47 smithi156 ceph-mon[22679]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:48.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:47 smithi070 ceph-mon[26293]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:48.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:47 smithi070 ceph-mon[21827]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:49.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:48 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:49.158 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:49.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:48 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:49.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:48 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:49 smithi156 ceph-mon[22679]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:50.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:49 smithi070 ceph-mon[26293]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:50.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:49 smithi070 ceph-mon[21827]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:52.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:51 smithi070 ceph-mon[26293]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:52.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:51 smithi070 ceph-mon[21827]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:51 smithi156 ceph-mon[22679]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:53.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:54.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:53 smithi070 ceph-mon[26293]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:54.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:53 smithi070 ceph-mon[21827]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:54.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:53 smithi156 ceph-mon[22679]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:56.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:55 smithi156 ceph-mon[22679]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:56.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:55 smithi070 ceph-mon[26293]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:56.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:55 smithi070 ceph-mon[21827]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:32:56.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:32:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:32:56] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-04-24T16:32:58.156 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:57 smithi070 ceph-mon[26293]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:58.156 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:57 smithi070 ceph-mon[21827]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:58.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:57 smithi156 ceph-mon[22679]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:32:58.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:58 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:58.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:32:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:32:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:32:59.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:58 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:32:59.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:58 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:59 smithi156 ceph-mon[22679]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:32:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:00.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:59 smithi070 ceph-mon[26293]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:00.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:32:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:00.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:59 smithi070 ceph-mon[21827]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:00.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:32:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:02.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:01 smithi156 ceph-mon[22679]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:02.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:01 smithi070 ceph-mon[26293]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:02.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:01 smithi070 ceph-mon[21827]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:03.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:04.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:03 smithi070 ceph-mon[26293]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:03 smithi070 ceph-mon[21827]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:04.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:03 smithi156 ceph-mon[22679]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:06.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:05 smithi156 ceph-mon[22679]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:06.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:05 smithi070 ceph-mon[26293]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:06.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:05 smithi070 ceph-mon[21827]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:06.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:33:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:33:06] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-04-24T16:33:08.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:07 smithi156 ceph-mon[22679]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:08.165 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:07 smithi070 ceph-mon[26293]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:08.165 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:07 smithi070 ceph-mon[21827]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:08.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:08 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:08.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:09.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:08 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:08 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:10.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:09 smithi156 ceph-mon[22679]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:10.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:09 smithi070 ceph-mon[26293]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:09 smithi070 ceph-mon[21827]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:12.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:11 smithi156 ceph-mon[22679]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:12.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:11 smithi070 ceph-mon[21827]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:11 smithi070 ceph-mon[26293]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:13.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:12 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:33:13.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:12 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:33:13.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:12 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:33:13.886 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:14.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:13 smithi070 ceph-mon[26293]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:14.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:13 smithi070 ceph-mon[21827]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:14.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:13 smithi156 ceph-mon[22679]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:15.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:14 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:15.222 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:14 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:15.222 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:14 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:15 smithi156 ceph-mon[22679]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:16.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:15 smithi070 ceph-mon[26293]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:16.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:15 smithi070 ceph-mon[21827]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:16.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:33:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:33:16] "GET /metrics HTTP/1.1" 200 37890 "" "Prometheus/2.33.4" 2024-04-24T16:33:18.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[26293]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:18.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:33:18.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:33:18.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[21827]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:33:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:33:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:18 smithi156 ceph-mon[22679]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:18.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:33:18.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:33:18.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:33:18.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:19 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:19.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:19 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:19.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:19 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:20.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:20 smithi070 ceph-mon[26293]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:20.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:20 smithi070 ceph-mon[21827]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:20.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:20 smithi156 ceph-mon[22679]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:22.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:22 smithi070 ceph-mon[26293]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:22.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:22 smithi070 ceph-mon[21827]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:22.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:22 smithi156 ceph-mon[22679]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:23.765 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:23 smithi070 ceph-mon[26293]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:23 smithi070 ceph-mon[21827]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:23 smithi156 ceph-mon[22679]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:25 smithi156 ceph-mon[22679]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:26.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:25 smithi070 ceph-mon[26293]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:26.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:25 smithi070 ceph-mon[21827]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:26.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:33:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:33:26] "GET /metrics HTTP/1.1" 200 37890 "" "Prometheus/2.33.4" 2024-04-24T16:33:28.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:27 smithi156 ceph-mon[22679]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:28.188 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:27 smithi070 ceph-mon[26293]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:28.188 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:27 smithi070 ceph-mon[21827]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:28.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:28 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:28.909 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:28 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:29.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:28 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:29 smithi156 ceph-mon[22679]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:29 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:29 smithi070 ceph-mon[26293]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:29 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:29 smithi070 ceph-mon[21827]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:30.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:29 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:32.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:31 smithi156 ceph-mon[22679]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:32.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:31 smithi070 ceph-mon[26293]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:32.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:31 smithi070 ceph-mon[21827]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:33.774 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:33 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:33 smithi070 ceph-mon[26293]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:34.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:33 smithi070 ceph-mon[21827]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:34.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:33 smithi156 ceph-mon[22679]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:36.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:35 smithi156 ceph-mon[22679]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:36.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:35 smithi070 ceph-mon[26293]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:36.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:35 smithi070 ceph-mon[21827]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:36.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:33:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:33:36] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-24T16:33:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:37 smithi156 ceph-mon[22679]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:38.199 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:37 smithi070 ceph-mon[26293]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:38.199 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:37 smithi070 ceph-mon[21827]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:38.908 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:38 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:38.908 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:38 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:39.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:38 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:39.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:38 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:40.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:39 smithi156 ceph-mon[22679]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:40.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:39 smithi070 ceph-mon[26293]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:40.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:39 smithi070 ceph-mon[21827]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:42.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:41 smithi156 ceph-mon[22679]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:42.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:41 smithi070 ceph-mon[21827]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:42.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:41 smithi070 ceph-mon[26293]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:43.774 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:43 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:43 smithi070 ceph-mon[26293]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:43 smithi070 ceph-mon[21827]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:44.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:43 smithi156 ceph-mon[22679]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:45.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:44 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:44 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:45.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:44 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:33:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:45 smithi156 ceph-mon[22679]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:46.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:45 smithi070 ceph-mon[26293]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:46.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:45 smithi070 ceph-mon[21827]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:46.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:33:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:33:46] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-04-24T16:33:48.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:47 smithi156 ceph-mon[22679]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:48.209 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:47 smithi070 ceph-mon[26293]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:48.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:47 smithi070 ceph-mon[21827]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:49.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:48 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:49.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:49.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:48 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:49.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:48 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:50.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:49 smithi156 ceph-mon[22679]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:50.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:49 smithi070 ceph-mon[21827]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:50.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:49 smithi070 ceph-mon[26293]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:52.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:51 smithi070 ceph-mon[26293]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:52.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:51 smithi070 ceph-mon[21827]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:52.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:51 smithi156 ceph-mon[22679]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:53.773 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi070\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi070\", job=\"node\", machine=\"x86_64\", nodename=\"smithi070\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:54.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:53 smithi070 ceph-mon[26293]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:54.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:53 smithi070 ceph-mon[21827]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:54.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:53 smithi156 ceph-mon[22679]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:56.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:55 smithi156 ceph-mon[22679]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:56.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:55 smithi070 ceph-mon[26293]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:56.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:55 smithi070 ceph-mon[21827]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:33:56.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:33:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:33:56] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-04-24T16:33:58.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:57 smithi156 ceph-mon[22679]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:58.218 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:57 smithi070 ceph-mon[26293]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:58.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:57 smithi070 ceph-mon[21827]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:33:59.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:58 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:59.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:33:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:33:58.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"172.21.15.156:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"be31be9a-0255-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.70\", device_class=\"ssd\", hostname=\"smithi070\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.70\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-24T16:33:59.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:58 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:33:59.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:58 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:59 smithi156 ceph-mon[22679]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:00.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:33:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:00.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:59 smithi070 ceph-mon[26293]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:00.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:33:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:00.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:59 smithi070 ceph-mon[21827]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:00.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:33:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:02.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:01 smithi156 ceph-mon[22679]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:02.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:01 smithi070 ceph-mon[26293]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:02.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:01 smithi070 ceph-mon[21827]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:04.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:03 smithi070 ceph-mon[21827]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:04.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:03 smithi070 ceph-mon[26293]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:04.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:03 smithi156 ceph-mon[22679]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:06.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:05 smithi156 ceph-mon[22679]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:06.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:05 smithi070 ceph-mon[26293]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:06.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:05 smithi070 ceph-mon[21827]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:06.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:34:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:34:06] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-04-24T16:34:08.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:07 smithi156 ceph-mon[22679]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:08.221 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:07 smithi070 ceph-mon[26293]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:08.221 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:07 smithi070 ceph-mon[21827]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:09.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:08 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:09.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:08 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:09.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:08 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:10.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:09 smithi156 ceph-mon[22679]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:10.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:09 smithi070 ceph-mon[26293]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:10.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:09 smithi070 ceph-mon[21827]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:12.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:11 smithi156 ceph-mon[22679]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:12.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:11 smithi070 ceph-mon[21827]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:12.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:11 smithi070 ceph-mon[26293]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:14.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:13 smithi070 ceph-mon[26293]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:14.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:13 smithi070 ceph-mon[21827]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:14.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:13 smithi156 ceph-mon[22679]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:15.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:14 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:15.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:14 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:15.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:14 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:16.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:15 smithi156 ceph-mon[22679]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:16.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:15 smithi070 ceph-mon[26293]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:16.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:15 smithi070 ceph-mon[21827]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:16.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:34:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:34:16] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-04-24T16:34:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:17 smithi156 ceph-mon[22679]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:17 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:34:18.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:17 smithi070 ceph-mon[26293]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:18.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:17 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:34:18.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:17 smithi070 ceph-mon[21827]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:18.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:17 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:34:19.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:18 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:19.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:34:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:18 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:34:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:18 smithi156 ceph-mon[22679]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:34:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:34:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:34:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[21827]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:34:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:34:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:34:19.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:18 smithi070 ceph-mon[26293]: from='mgr.34682 ' entity='mgr.x' 2024-04-24T16:34:20.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:20 smithi156 ceph-mon[22679]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:20.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:20 smithi070 ceph-mon[26293]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:20.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:20 smithi070 ceph-mon[21827]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:22.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:22 smithi156 ceph-mon[22679]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:22.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:22 smithi070 ceph-mon[26293]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:22.791 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:22 smithi070 ceph-mon[21827]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:24.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:23 smithi070 ceph-mon[26293]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:24.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:23 smithi070 ceph-mon[21827]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:24.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:23 smithi156 ceph-mon[22679]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:26.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:25 smithi156 ceph-mon[22679]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:26.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:25 smithi070 ceph-mon[26293]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:26.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:25 smithi070 ceph-mon[21827]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:26.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:34:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:34:26] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-04-24T16:34:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:27 smithi156 ceph-mon[22679]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:28.243 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:27 smithi070 ceph-mon[26293]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:28.243 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:27 smithi070 ceph-mon[21827]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:29.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:28 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:28 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:29.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:28 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:29 smithi156 ceph-mon[22679]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:29 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:29 smithi070 ceph-mon[26293]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:29 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:30.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:29 smithi070 ceph-mon[21827]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:30.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:29 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:32.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:31 smithi070 ceph-mon[26293]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:32.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:31 smithi070 ceph-mon[21827]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:31 smithi156 ceph-mon[22679]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:34.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:33 smithi070 ceph-mon[26293]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:34.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:33 smithi070 ceph-mon[21827]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:34.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:33 smithi156 ceph-mon[22679]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:36.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:35 smithi156 ceph-mon[22679]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:36.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:35 smithi070 ceph-mon[26293]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:36.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:35 smithi070 ceph-mon[21827]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:36.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:34:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:34:36] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-04-24T16:34:38.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:37 smithi156 ceph-mon[22679]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:38.250 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:37 smithi070 ceph-mon[26293]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:38.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:37 smithi070 ceph-mon[21827]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:39.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:38 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:39.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:38 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:39.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:38 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:40.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:39 smithi070 ceph-mon[26293]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:40.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:39 smithi070 ceph-mon[21827]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:40.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:39 smithi156 ceph-mon[22679]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:42.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:41 smithi070 ceph-mon[21827]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:42.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:41 smithi070 ceph-mon[26293]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:42.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:41 smithi156 ceph-mon[22679]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:44.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:43 smithi070 ceph-mon[21827]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:44.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:43 smithi070 ceph-mon[26293]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:44.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:43 smithi156 ceph-mon[22679]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:45.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:44 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:45.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:44 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:45.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:44 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:34:46.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:45 smithi156 ceph-mon[22679]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:46.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:45 smithi070 ceph-mon[26293]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:46.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:45 smithi070 ceph-mon[21827]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:46.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:34:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:34:46] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-24T16:34:48.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:47 smithi156 ceph-mon[22679]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:48.259 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:47 smithi070 ceph-mon[26293]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:48.259 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:47 smithi070 ceph-mon[21827]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:49.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:48 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:49.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:48 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:49.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:48 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:50.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:49 smithi070 ceph-mon[21827]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:50.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:49 smithi070 ceph-mon[26293]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:50.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:49 smithi156 ceph-mon[22679]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:52.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:51 smithi070 ceph-mon[26293]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:52.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:51 smithi070 ceph-mon[21827]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:51 smithi156 ceph-mon[22679]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:54.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:53 smithi070 ceph-mon[21827]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:54.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:53 smithi070 ceph-mon[26293]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:54.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:53 smithi156 ceph-mon[22679]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:56.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:55 smithi156 ceph-mon[22679]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:56.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:55 smithi070 ceph-mon[26293]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:56.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:55 smithi070 ceph-mon[21827]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:34:56.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:34:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:34:56] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-24T16:34:57.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:34:58.270 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:57 smithi070 ceph-mon[26293]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:58.271 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:57 smithi070 ceph-mon[21827]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:58.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:57 smithi156 ceph-mon[22679]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:34:59.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:58 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:59.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:58 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:59.330 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:34:59.330 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (5m) 102s ago 13m 24.6M - 0.23.0 ba2b418f427c 609b591dc19c 2024-04-24T16:34:59.330 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (5m) 3m ago 13m 54.6M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:34:59.330 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (2m) 102s ago 12m 50.5M - 3.5 e1d6a67b021e 662cf6a079f3 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283 running (8m) 3m ago 17m 511M - 19.0.0-3285-g5904d294 8bb2087275ff 432b981bb958 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (2m) 102s ago 19m 452M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (19m) 102s ago 19m 57.3M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (18m) 3m ago 18m 50.0M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (18m) 102s ago 18m 44.6M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (6m) 102s ago 14m 16.5M - 1.3.1 1dbe0e931976 a0dadc7082e4 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (5m) 3m ago 14m 15.6M - 1.3.1 1dbe0e931976 cda127a7d141 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (17m) 102s ago 17m 55.1M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (17m) 102s ago 17m 56.1M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (16m) 102s ago 16m 50.0M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (16m) 102s ago 16m 50.1M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (16m) 3m ago 16m 50.7M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (15m) 3m ago 15m 50.3M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:34:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (15m) 3m ago 15m 47.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:34:59.332 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (15m) 3m ago 15m 52.2M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:34:59.332 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (5m) 3m ago 14m 70.5M - 2.33.4 514e6a882f6e 0e9a48e5d60e 2024-04-24T16:34:59.332 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (12m) 102s ago 12m 72.4M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:34:59.332 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (12m) 3m ago 12m 71.9M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:34:59.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:58 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:34:59.966 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions' 2024-04-24T16:35:00.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:59 smithi070 ceph-mon[26293]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:00.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:59 smithi070 ceph-mon[26293]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:35:00.268 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:34:59 smithi070 ceph-mon[26293]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:00.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:59 smithi070 ceph-mon[21827]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:00.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:59 smithi070 ceph-mon[21827]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:35:00.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:34:59 smithi070 ceph-mon[21827]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:59 smithi156 ceph-mon[22679]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:59 smithi156 ceph-mon[22679]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:35:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:34:59 smithi156 ceph-mon[22679]: from='mgr.34682 172.21.15.156:0/2735647469' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:02.021 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:35:02.021 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:35:02.021 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:35:02.021 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:35:02.021 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:35:02.021 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:35:02.022 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:35:02.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:01 smithi070 ceph-mon[26293]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:02.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:01 smithi070 ceph-mon[21827]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:02.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:01 smithi156 ceph-mon[22679]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:02.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:35:03.171 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:02 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1209849166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:35:03.172 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:02 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1209849166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:35:03.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:02 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1209849166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:35:04.176 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:03 smithi070 ceph-mon[26293]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:04.176 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:03 smithi070 ceph-mon[21827]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:04.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:03 smithi156 ceph-mon[22679]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:04.702 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_OK 2024-04-24T16:35:05.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:04 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3149384693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:35:05.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:04 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3149384693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:35:05.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph -s' 2024-04-24T16:35:05.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:04 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3149384693' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:35:05.941 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:05 smithi070 ceph-mon[21827]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:05.941 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:05 smithi070 ceph-mon[26293]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:06.330 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:05 smithi156 ceph-mon[22679]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:06.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:35:06] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-04-24T16:35:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: cluster: 2024-04-24T16:35:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: id: be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:35:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: health: HEALTH_OK 2024-04-24T16:35:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: services: 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: mgr: x(active, since 6m), standbys: y 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: data: 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: pools: 6 pools, 161 pgs 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: objects: 209 objects, 585 KiB 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: usage: 90 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: pgs: 161 active+clean 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: io: 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-04-24T16:35:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:35:08.135 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:07 smithi070 ceph-mon[26293]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:08.135 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:07 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2074081941' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:35:08.136 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:07 smithi070 ceph-mon[21827]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:08.136 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:07 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2074081941' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:35:08.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph mgr fail' 2024-04-24T16:35:08.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:07 smithi156 ceph-mon[22679]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:07 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2074081941' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:35:09.041 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:08 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:09.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:08 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:09.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:08 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:10.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:09 smithi070 ceph-mon[21827]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:10.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:09 smithi070 ceph-mon[26293]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:10.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:09 smithi156 ceph-mon[22679]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:11.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3735368811' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:35:11.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-mon[26293]: osdmap e103: 8 total, 8 up, 8 in 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3735368811' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-mon[21827]: osdmap e103: 8 total, 8 up, 8 in 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:10] ENGINE Bus STOPPING 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:10 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:10] ENGINE Bus STOPPED 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:11] ENGINE Bus STARTING 2024-04-24T16:35:11.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:11] ENGINE Serving on http://:::9283 2024-04-24T16:35:11.293 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:11] ENGINE Bus STARTED 2024-04-24T16:35:11.317 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:10 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3735368811' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:10 smithi156 ceph-mon[22679]: osdmap e103: 8 total, 8 up, 8 in 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:10 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:10.948+0000 7f86100f8700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ignoring --setuser ceph since I am not root 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ignoring --setgroup ceph since I am not root 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mgr[42890]: -- 172.21.15.156:0/3345168416 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648ea17e5a0 con 0x5648ea153000 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:11.126+0000 7fe9ff616200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:35:11.318 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:11.319+0000 7fe9ff616200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:35:11.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:11.427+0000 7fe9ff616200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:35:11.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'sleep 180' 2024-04-24T16:35:12.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3735368811' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:35:12.114 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: mgrmap e28: y(active, starting, since 0.851815s), standbys: x 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:35:12.115 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: Manager daemon y is now available 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:35:12.116 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:11.841Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.156:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.156:8765: connect: connection refused" 2024-04-24T16:35:12.117 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:11.842Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.156:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.156:8765: connect: connection refused" 2024-04-24T16:35:12.117 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:11.843Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.156:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.156:8765: connect: connection refused" 2024-04-24T16:35:12.117 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:11.844Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.156:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.156:8765: connect: connection refused" 2024-04-24T16:35:12.117 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:11.844Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.156:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.156:8765: connect: connection refused" 2024-04-24T16:35:12.117 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:11.752+0000 7fe9ff616200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:35:12.117 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:11.849+0000 7fe9ff616200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:35:12.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3735368811' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:35:12.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: mgrmap e28: y(active, starting, since 0.851815s), standbys: x 2024-04-24T16:35:12.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:35:12.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:35:12.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:35:12.282 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: Manager daemon y is now available 2024-04-24T16:35:12.283 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3735368811' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: mgrmap e28: y(active, starting, since 0.851815s), standbys: x 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:35:12.284 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: Manager daemon y is now available 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:12.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:35:12.286 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:35:12.286 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:35:12.286 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:11 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:35:13.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[21827]: [24/Apr/2024:16:35:11] ENGINE Bus STARTING 2024-04-24T16:35:13.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[21827]: [24/Apr/2024:16:35:11] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:35:13.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[21827]: [24/Apr/2024:16:35:11] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:35:13.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[21827]: [24/Apr/2024:16:35:11] ENGINE Bus STARTED 2024-04-24T16:35:13.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[21827]: mgrmap e29: y(active, since 1.85872s), standbys: x 2024-04-24T16:35:13.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[21827]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[26293]: [24/Apr/2024:16:35:11] ENGINE Bus STARTING 2024-04-24T16:35:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[26293]: [24/Apr/2024:16:35:11] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:35:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[26293]: [24/Apr/2024:16:35:11] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:35:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[26293]: [24/Apr/2024:16:35:11] ENGINE Bus STARTED 2024-04-24T16:35:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[26293]: mgrmap e29: y(active, since 1.85872s), standbys: x 2024-04-24T16:35:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:12 smithi070 ceph-mon[26293]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:13.159 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:13.102+0000 7fe9ff616200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:35:13.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:12 smithi156 ceph-mon[22679]: [24/Apr/2024:16:35:11] ENGINE Bus STARTING 2024-04-24T16:35:13.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:12 smithi156 ceph-mon[22679]: [24/Apr/2024:16:35:11] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:35:13.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:12 smithi156 ceph-mon[22679]: [24/Apr/2024:16:35:11] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:35:13.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:12 smithi156 ceph-mon[22679]: [24/Apr/2024:16:35:11] ENGINE Bus STARTED 2024-04-24T16:35:13.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:12 smithi156 ceph-mon[22679]: mgrmap e29: y(active, since 1.85872s), standbys: x 2024-04-24T16:35:13.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:12 smithi156 ceph-mon[22679]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:13.459 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:13.277+0000 7fe9ff616200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:35:13.459 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:13.459+0000 7fe9ff616200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:35:14.094 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:13 smithi156 ceph-mon[22679]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:14.094 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:13 smithi156 ceph-mon[22679]: mgrmap e30: y(active, since 2s), standbys: x 2024-04-24T16:35:14.094 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:13.775+0000 7fe9ff616200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:35:14.094 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:13.887+0000 7fe9ff616200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:35:14.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:13 smithi070 ceph-mon[21827]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:14.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:13 smithi070 ceph-mon[21827]: mgrmap e30: y(active, since 2s), standbys: x 2024-04-24T16:35:14.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:13 smithi070 ceph-mon[26293]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:14.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:13 smithi070 ceph-mon[26293]: mgrmap e30: y(active, since 2s), standbys: x 2024-04-24T16:35:14.359 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:14.095+0000 7fe9ff616200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:35:14.359 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:14.310+0000 7fe9ff616200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:35:15.329 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:14.976+0000 7fe9ff616200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:35:15.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:15.451+0000 7fe9ff616200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:35:15.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:15.548+0000 7fe9ff616200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:35:16.159 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:15.770+0000 7fe9ff616200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:35:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:15 smithi156 ceph-mon[22679]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:15 smithi156 ceph-mon[22679]: mgrmap e31: y(active, since 4s), standbys: x 2024-04-24T16:35:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:15 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:16.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:15 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:16.269 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[26293]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:16.269 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[26293]: mgrmap e31: y(active, since 4s), standbys: x 2024-04-24T16:35:16.269 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:16.269 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:16.270 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[21827]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:16.270 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[21827]: mgrmap e31: y(active, since 4s), standbys: x 2024-04-24T16:35:16.270 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:16.270 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:15 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:16.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:16.324+0000 7fe9ff616200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:35:16.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:16.422+0000 7fe9ff616200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:35:17.050 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:16.684+0000 7fe9ff616200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:35:17.051 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:16.828+0000 7fe9ff616200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:35:17.051 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:16.931+0000 7fe9ff616200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:35:17.051 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:17.051+0000 7fe9ff616200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:35:17.658 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:17.361+0000 7fe9ff616200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:35:18.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-mon[22679]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:35:18.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.159 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:17.974+0000 7fe9ff616200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:35:18.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[26293]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[21827]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.289 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:17 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:18.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:18.192+0000 7fe9ff616200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:35:18.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:35:18] ENGINE Bus STARTING 2024-04-24T16:35:18.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CherryPy Checker: 2024-04-24T16:35:18.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: The Application mounted at '' has an empty config. 2024-04-24T16:35:18.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:35:18.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:35:18] ENGINE Serving on http://:::9283 2024-04-24T16:35:18.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:35:18] ENGINE Bus STARTED 2024-04-24T16:35:19.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: Standby manager daemon x restarted 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:18 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:19.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: Standby manager daemon x restarted 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: Standby manager daemon x restarted 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:35:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='client.15249 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:18 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-24T16:35:20.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:19 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:20.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:35:20.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:35:20.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:35:20.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:35:20.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:20.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:20.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:20.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:20.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:19 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:21.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:20 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:35:21.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:20 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:35:21.196 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:20 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:35:21.196 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:20 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:35:21.196 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:20 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:35:21.196 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:20 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:35:22.041 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:21 smithi070 ceph-mon[26293]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:35:22.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:21 smithi070 ceph-mon[21827]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:35:22.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:21 smithi156 ceph-mon[22679]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:35:22.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:35:22.592+0000 7f8c16aea700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:22.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Creating ceph-iscsi config... 2024-04-24T16:35:22.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:22.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:22.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:22.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Traceback (most recent call last): 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: return _run_code(code, main_globals, None, 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: exec(code, run_globals) 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:22.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:22.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:22.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Traceback (most recent call last): 2024-04-24T16:35:22.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:35:22.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:35:22.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:35:22.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: return future.result(timeout) 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: return self.__get_result() 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: raise self._exception 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: use_current_daemon_image=reconfig, 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:22.795 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Creating ceph-iscsi config... 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: Traceback (most recent call last): 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: return _run_code(code, main_globals, None, 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: exec(code, run_globals) 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:22.796 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:22.797 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:22.797 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:22.797 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:22.797 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:22.797 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:22 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.042 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:35:24.044 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:24.045 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:35:24.048 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:24.049 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[21827]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:35:24.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[21827]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[21827]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[26293]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[26293]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:35:24.051 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:23 smithi070 ceph-mon[26293]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:35:24.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:23 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:24.159 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:35:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:35:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:35:24.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:35:24.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:35:24.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:35:24.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:23 smithi156 ceph-mon[22679]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:35:24.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:23 smithi156 ceph-mon[22679]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:35:24.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:23 smithi156 ceph-mon[22679]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:35:26.203 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-mon[22679]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:26.204 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 systemd[1]: Stopping Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:35:26.204 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 bash[51395]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:35:26.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:26 smithi070 ceph-mon[21827]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:26.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:26 smithi070 ceph-mon[26293]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.205Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.205Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.205Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.205Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.205Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.205Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.205Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.206Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-24T16:35:26.496 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.206Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-24T16:35:26.497 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.206Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-24T16:35:26.497 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.206Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-24T16:35:26.497 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[46049]: ts=2024-04-24T16:35:26.206Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-24T16:35:26.497 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 podman[51408]: 2024-04-24 16:35:26.215620902 +0000 UTC m=+0.112203355 container died 0e9a48e5d60edc83024f412c27eae570cdc3b7861526196dac61ba516a2256eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:35:26.871 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 podman[51408]: 2024-04-24 16:35:26.588287684 +0000 UTC m=+0.484870135 container cleanup 0e9a48e5d60edc83024f412c27eae570cdc3b7861526196dac61ba516a2256eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:35:26.871 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 bash[51408]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a 2024-04-24T16:35:26.871 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 bash[51435]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:35:27.121 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:27.122 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 podman[51422]: 2024-04-24 16:35:26.873208545 +0000 UTC m=+0.659731124 container remove 0e9a48e5d60edc83024f412c27eae570cdc3b7861526196dac61ba516a2256eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:35:27.122 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-24T16:35:27.122 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 systemd[1]: Stopped Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:35:27.122 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 2.028s CPU time. 2024-04-24T16:35:27.122 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:26 smithi156 systemd[1]: Starting Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:35:27.122 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 podman[51507]: 2024-04-24 16:35:27.013151671 +0000 UTC m=+0.019160832 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-24T16:35:27.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:27.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:27.383 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 podman[51507]: 2024-04-24 16:35:27.124009805 +0000 UTC m=+0.130018954 container create 072ede669133e8e8520287378724307979c0b98476e8589d206d2f83c2eacb13 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:35:27.384 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 podman[51507]: 2024-04-24 16:35:27.359964495 +0000 UTC m=+0.365973647 container init 072ede669133e8e8520287378724307979c0b98476e8589d206d2f83c2eacb13 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:35:27.384 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 podman[51507]: 2024-04-24 16:35:27.362565208 +0000 UTC m=+0.368574360 container start 072ede669133e8e8520287378724307979c0b98476e8589d206d2f83c2eacb13 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.385Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.385Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.385Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.385Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi156 (none))" 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.385Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.386Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.387Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.388Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-24T16:35:27.659 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.388Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-24T16:35:27.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.390Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-24T16:35:27.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.390Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.135µs 2024-04-24T16:35:27.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.390Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-24T16:35:27.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 bash[51507]: 072ede669133e8e8520287378724307979c0b98476e8589d206d2f83c2eacb13 2024-04-24T16:35:27.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 systemd[1]: Started Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:35:27.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.597Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-04-24T16:35:27.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:27] ENGINE Bus STOPPING 2024-04-24T16:35:27.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:35:27.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:27] ENGINE Bus STOPPED 2024-04-24T16:35:27.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:27] ENGINE Bus STARTING 2024-04-24T16:35:27.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:27] ENGINE Serving on http://:::9283 2024-04-24T16:35:27.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:35:27] ENGINE Bus STARTED 2024-04-24T16:35:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-mon[22679]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:35:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:35:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:28.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:27.826Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.032Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.032Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.032Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=48.386µs wal_replay_duration=642.808583ms total_replay_duration=642.877637ms 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.037Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.037Z caller=main.go:947 level=info msg="TSDB started" 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.037Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.064Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=26.762286ms db_storage=816ns remote_storage=1.813µs web_handler=439ns query_engine=860ns scrape=229.169µs scrape_sd=117.188µs notify=10.531µs notify_sd=10.516µs rules=26.042754ms 2024-04-24T16:35:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:35:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:35:28.064Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[26293]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[21827]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:28.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:35:28.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:35:28.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:28.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:28 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:29.091 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:29 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:35:29.091 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:29 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:35:29.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:29 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:35:29.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:29 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:35:29.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:29 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:35:29.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:29 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:35:30.158 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:30 smithi156 ceph-mon[22679]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:30.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:30 smithi070 ceph-mon[26293]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:30.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:30 smithi070 ceph-mon[21827]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:35:32.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:32 smithi070 ceph-mon[26293]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:35:32.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:32 smithi070 ceph-mon[21827]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:35:32.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:32 smithi156 ceph-mon[22679]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:35:33.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:33.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:35:33.351 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:33.351 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:33.351 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:33.351 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:33 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:35:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:35:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:35:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:33 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:35:34.379 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:34 smithi070 ceph-mon[26293]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:35:34.379 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:34 smithi070 ceph-mon[21827]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:35:34.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:34 smithi156 ceph-mon[22679]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:35:36.159 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:36 smithi070 ceph-mon[26293]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:36.159 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:36 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/390081542' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:35:36.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:36 smithi070 ceph-mon[21827]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:36.160 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:36 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/390081542' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:35:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:36 smithi156 ceph-mon[22679]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:35:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:36 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/390081542' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:35:36.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:35:36] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2853135401' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:38 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2853135401' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:35:38.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:38 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2853135401' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:35:40.534 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:40 smithi070 ceph-mon[26293]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:40.534 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:40 smithi070 ceph-mon[21827]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:40.658 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:40 smithi156 ceph-mon[22679]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:41.457 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:41 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:41.457 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:41 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:41.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:41 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:42.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[21827]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:42.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:42.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:42.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[26293]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:42 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:42 smithi156 ceph-mon[22679]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-24T16:35:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:42 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:42 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:42 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:35:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:42 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:35:42.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:42 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:35:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:43 smithi156 ceph-mon[22679]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-24T16:35:44.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:43 smithi070 ceph-mon[26293]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-24T16:35:44.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:43 smithi070 ceph-mon[21827]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-24T16:35:46.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:46 smithi070 ceph-mon[26293]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-24T16:35:46.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:46 smithi070 ceph-mon[21827]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-24T16:35:46.311 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:46 smithi156 ceph-mon[22679]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-24T16:35:46.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:35:46] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-04-24T16:35:48.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:48 smithi070 ceph-mon[26293]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-24T16:35:48.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:48 smithi070 ceph-mon[21827]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-24T16:35:48.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:48 smithi156 ceph-mon[22679]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-24T16:35:49.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:49 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:49.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:49 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:49.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:49 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:50.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:50 smithi070 ceph-mon[26293]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:50.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:50 smithi070 ceph-mon[21827]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:50.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:50 smithi156 ceph-mon[22679]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:52.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:52 smithi070 ceph-mon[26293]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:52.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:52 smithi070 ceph-mon[21827]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:52 smithi156 ceph-mon[22679]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:54.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:53 smithi156 ceph-mon[22679]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:54.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:53 smithi070 ceph-mon[26293]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:54.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:53 smithi070 ceph-mon[21827]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:56.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:56 smithi156 ceph-mon[22679]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:56.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:56 smithi070 ceph-mon[26293]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:56.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:56 smithi070 ceph-mon[21827]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:35:56.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:35:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:35:56] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-04-24T16:35:57.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:57 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:57.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:57 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:57.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:57 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:35:58.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:58 smithi070 ceph-mon[26293]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:58.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:58 smithi070 ceph-mon[21827]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:58.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:58 smithi156 ceph-mon[22679]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:35:59.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:35:59 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:59.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:35:59 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:35:59.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:35:59 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:00.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:00 smithi070 ceph-mon[26293]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:00.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:00 smithi070 ceph-mon[21827]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:00 smithi156 ceph-mon[22679]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:02.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:02 smithi070 ceph-mon[26293]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:02.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:02 smithi070 ceph-mon[21827]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:02.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:02 smithi156 ceph-mon[22679]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:04.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:03 smithi156 ceph-mon[22679]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:04.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:03 smithi070 ceph-mon[26293]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:04.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:03 smithi070 ceph-mon[21827]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:06.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:06 smithi070 ceph-mon[26293]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:06.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:06 smithi070 ceph-mon[21827]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:06.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:06 smithi156 ceph-mon[22679]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:06.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:36:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:36:06] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-04-24T16:36:08.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:08 smithi070 ceph-mon[26293]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:08.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:08 smithi070 ceph-mon[21827]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:08.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:08 smithi156 ceph-mon[22679]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:09 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:09.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:09 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:09.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:09 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:10.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:10 smithi070 ceph-mon[26293]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:10.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:10 smithi070 ceph-mon[21827]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:10.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:10 smithi156 ceph-mon[22679]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:12 smithi070 ceph-mon[21827]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:12.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:12 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:12.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:12 smithi070 ceph-mon[26293]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:12.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:12 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:12.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:12 smithi156 ceph-mon[22679]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:12.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:12 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:14.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:13 smithi156 ceph-mon[22679]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:14.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:13 smithi070 ceph-mon[21827]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:14.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:13 smithi070 ceph-mon[26293]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:16.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:16 smithi070 ceph-mon[21827]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:16.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:16 smithi070 ceph-mon[26293]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:16.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:16 smithi156 ceph-mon[22679]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:16.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:36:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:36:16] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-04-24T16:36:18.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:18 smithi070 ceph-mon[26293]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:18 smithi070 ceph-mon[21827]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:18.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:18 smithi156 ceph-mon[22679]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:19 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:19 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:19.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:19 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:20.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:20 smithi070 ceph-mon[21827]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:20.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:20 smithi070 ceph-mon[26293]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:20.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:20 smithi156 ceph-mon[22679]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:22.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:22 smithi070 ceph-mon[21827]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:22.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:22 smithi070 ceph-mon[26293]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:22.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:22 smithi156 ceph-mon[22679]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:24.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:23 smithi156 ceph-mon[22679]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:24.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:23 smithi070 ceph-mon[21827]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:24.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:23 smithi070 ceph-mon[26293]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:26.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:26 smithi070 ceph-mon[26293]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:26.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:26 smithi070 ceph-mon[21827]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:26.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:26 smithi156 ceph-mon[22679]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:26.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:36:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:36:26] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-04-24T16:36:27.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:27 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:27.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:27 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:27.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:27 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:28.291 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:28 smithi070 ceph-mon[26293]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:28.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:28 smithi070 ceph-mon[21827]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:28.408 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:28 smithi156 ceph-mon[22679]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:29.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:29 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:29.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:29 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:29.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:29 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:30.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:30 smithi156 ceph-mon[22679]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:30.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:30 smithi070 ceph-mon[26293]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:30.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:30 smithi070 ceph-mon[21827]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:32 smithi156 ceph-mon[22679]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:32.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:32 smithi070 ceph-mon[26293]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:32.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:32 smithi070 ceph-mon[21827]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:34.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:33 smithi156 ceph-mon[22679]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:34.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:33 smithi070 ceph-mon[26293]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:34.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:33 smithi070 ceph-mon[21827]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:36.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:36 smithi070 ceph-mon[26293]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:36.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:36 smithi070 ceph-mon[21827]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:36 smithi156 ceph-mon[22679]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:36.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:36:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:36:36] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-04-24T16:36:38.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:38 smithi070 ceph-mon[21827]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:38.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:38 smithi070 ceph-mon[26293]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:38.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:38 smithi156 ceph-mon[22679]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:39.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:39 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:39.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:39 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:39.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:39 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:40.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:40 smithi156 ceph-mon[22679]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:40.541 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:40 smithi070 ceph-mon[21827]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:40.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:40 smithi070 ceph-mon[26293]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:41.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:41 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:41.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:41 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:41.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:41 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:42.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:42 smithi156 ceph-mon[22679]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:42.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:42 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:36:42.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:42 smithi070 ceph-mon[26293]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:42.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:42 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:36:42.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:42 smithi070 ceph-mon[21827]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:42.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:42 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:36:43.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:43 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:43 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:43 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:43 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:43 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:36:43.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:43 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:36:43.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:43 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:36:43.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:36:43.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:43.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:36:43.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:36:43.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:43 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:36:44.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:44 smithi156 ceph-mon[22679]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:44.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:44 smithi070 ceph-mon[26293]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:44.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:44 smithi070 ceph-mon[21827]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:46.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:46 smithi156 ceph-mon[22679]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:46.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:46 smithi070 ceph-mon[26293]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:46.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:36:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:36:46] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-04-24T16:36:46.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:46 smithi070 ceph-mon[21827]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:48.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:48 smithi156 ceph-mon[22679]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:48.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:48 smithi070 ceph-mon[26293]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:48.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:48 smithi070 ceph-mon[21827]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:49.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:49 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:49.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:49 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:49.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:49 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:50.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:50 smithi156 ceph-mon[22679]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:50.791 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:50 smithi070 ceph-mon[26293]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:50.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:50 smithi070 ceph-mon[21827]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:52.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:52 smithi156 ceph-mon[22679]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:52.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:52 smithi070 ceph-mon[26293]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:52.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:52 smithi070 ceph-mon[21827]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:54.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:53 smithi156 ceph-mon[22679]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:54.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:53 smithi070 ceph-mon[26293]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:54.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:53 smithi070 ceph-mon[21827]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:56.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:56 smithi070 ceph-mon[26293]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:56.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:56 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:56.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:56 smithi070 ceph-mon[21827]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:56.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:56 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:56 smithi156 ceph-mon[22679]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:36:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:56 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:36:56.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:36:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:36:56] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-04-24T16:36:58.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:58 smithi156 ceph-mon[22679]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:58 smithi070 ceph-mon[26293]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:58.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:58 smithi070 ceph-mon[21827]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:36:59.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:36:59 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:59.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:36:59 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:36:59.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:36:59 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:00 smithi156 ceph-mon[22679]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:00.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:00 smithi070 ceph-mon[26293]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:00.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:00 smithi070 ceph-mon[21827]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:02.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:02 smithi156 ceph-mon[22679]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:02.541 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:02 smithi070 ceph-mon[26293]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:02.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:02 smithi070 ceph-mon[21827]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:04.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:03 smithi156 ceph-mon[22679]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:04.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:03 smithi070 ceph-mon[26293]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:04.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:03 smithi070 ceph-mon[21827]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:06.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:06 smithi156 ceph-mon[22679]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:06.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:06 smithi070 ceph-mon[26293]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:06.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:06 smithi070 ceph-mon[21827]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:06.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:37:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:37:06] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-04-24T16:37:08.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:08 smithi156 ceph-mon[22679]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:08 smithi070 ceph-mon[26293]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:08.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:08 smithi070 ceph-mon[21827]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:09.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:09 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:09.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:09 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:09.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:09 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:10.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:10 smithi156 ceph-mon[22679]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:10.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:10 smithi070 ceph-mon[26293]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:10.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:10 smithi070 ceph-mon[21827]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:11.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:11.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:11.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:12.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:12 smithi156 ceph-mon[22679]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:12.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:12 smithi070 ceph-mon[26293]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:12.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:12 smithi070 ceph-mon[21827]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:14.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:13 smithi156 ceph-mon[22679]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:14.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:13 smithi070 ceph-mon[26293]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:14.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:13 smithi070 ceph-mon[21827]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:16.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:16 smithi156 ceph-mon[22679]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:16.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:16 smithi070 ceph-mon[26293]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:16.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:16 smithi070 ceph-mon[21827]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:16.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:37:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:37:16] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-04-24T16:37:18.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:18 smithi156 ceph-mon[22679]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:18.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:18 smithi070 ceph-mon[21827]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:18.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:18 smithi070 ceph-mon[26293]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:19.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:19 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:19.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:19 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:19.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:19 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:20.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:20 smithi156 ceph-mon[22679]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:20.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:20 smithi070 ceph-mon[26293]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:20.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:20 smithi070 ceph-mon[21827]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:22.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:22 smithi156 ceph-mon[22679]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:22.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:22 smithi070 ceph-mon[26293]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:22.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:22 smithi070 ceph-mon[21827]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:24.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:23 smithi156 ceph-mon[22679]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:24.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:23 smithi070 ceph-mon[26293]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:24.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:23 smithi070 ceph-mon[21827]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:26.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:26 smithi156 ceph-mon[22679]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:26.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:26 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:26.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:26 smithi070 ceph-mon[21827]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:26.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:26 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:26.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:26 smithi070 ceph-mon[26293]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:26.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:26 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:26.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:37:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:37:26] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-04-24T16:37:28.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:28 smithi156 ceph-mon[22679]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:28.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:28 smithi070 ceph-mon[26293]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:28.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:28 smithi070 ceph-mon[21827]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:29.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:29 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:29.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:29 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:29.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:29 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:30.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:30 smithi156 ceph-mon[22679]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:30.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:30 smithi070 ceph-mon[26293]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:30.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:30 smithi070 ceph-mon[21827]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:32 smithi156 ceph-mon[22679]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:32.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:32 smithi070 ceph-mon[26293]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:32.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:32 smithi070 ceph-mon[21827]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:34.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:33 smithi156 ceph-mon[22679]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:34.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:33 smithi070 ceph-mon[26293]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:34.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:33 smithi070 ceph-mon[21827]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:36 smithi156 ceph-mon[22679]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:36.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:36 smithi070 ceph-mon[26293]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:36.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:36 smithi070 ceph-mon[21827]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:36.791 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:37:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:37:36] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-04-24T16:37:38.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:38 smithi156 ceph-mon[22679]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:38.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:38 smithi070 ceph-mon[26293]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:38.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:38 smithi070 ceph-mon[21827]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:39.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:39 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:39.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:39 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:39.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:39 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:40.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:40 smithi156 ceph-mon[22679]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:40.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:40 smithi070 ceph-mon[21827]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:40.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:40 smithi070 ceph-mon[26293]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:41.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:41 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:41.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:41 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:41.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:41 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:42.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:42 smithi156 ceph-mon[22679]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:42.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:42 smithi070 ceph-mon[21827]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:42.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:42 smithi070 ceph-mon[26293]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:43 smithi156 ceph-mon[22679]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:43 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:37:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:43 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:37:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:43 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:37:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:43 smithi156 ceph-mon[22679]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:37:44.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[21827]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:44.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:37:44.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:37:44.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:37:44.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[21827]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:37:44.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[26293]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:44.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:37:44.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:37:44.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:37:44.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:43 smithi070 ceph-mon[26293]: from='mgr.25120 ' entity='mgr.y' 2024-04-24T16:37:46.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:46 smithi156 ceph-mon[22679]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:46.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:46 smithi070 ceph-mon[26293]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:46.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:46 smithi070 ceph-mon[21827]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:46.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:37:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:37:46] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-04-24T16:37:48.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:48 smithi156 ceph-mon[22679]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:48.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:48 smithi070 ceph-mon[26293]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:48.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:48 smithi070 ceph-mon[21827]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:49.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:49 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:49.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:49 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:49.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:49 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:50.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:50 smithi156 ceph-mon[22679]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:50.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:50 smithi070 ceph-mon[26293]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:50.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:50 smithi070 ceph-mon[21827]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:52.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:52 smithi156 ceph-mon[22679]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:52.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:52 smithi070 ceph-mon[21827]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:52.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:52 smithi070 ceph-mon[26293]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:54.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:53 smithi156 ceph-mon[22679]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:54.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:53 smithi070 ceph-mon[26293]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:54.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:53 smithi070 ceph-mon[21827]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:56.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:56 smithi156 ceph-mon[22679]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:56.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:56 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:56.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:56 smithi070 ceph-mon[26293]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:56.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:56 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:56.519 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:56 smithi070 ceph-mon[21827]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:37:56.519 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:56 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:37:56.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:37:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:37:56] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-04-24T16:37:58.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:58 smithi156 ceph-mon[22679]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:58.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:58 smithi070 ceph-mon[26293]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:58.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:58 smithi070 ceph-mon[21827]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:37:59.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:37:59 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:59.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:37:59 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:37:59.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:37:59 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:00 smithi156 ceph-mon[22679]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:00.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:00 smithi070 ceph-mon[21827]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:00.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:00 smithi070 ceph-mon[26293]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:02.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:02 smithi156 ceph-mon[22679]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:02.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:02 smithi070 ceph-mon[21827]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:02.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:02 smithi070 ceph-mon[26293]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:04.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:03 smithi156 ceph-mon[22679]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:04.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:03 smithi070 ceph-mon[21827]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:04.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:03 smithi070 ceph-mon[26293]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:06.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:06 smithi156 ceph-mon[22679]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:06.518 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:06 smithi070 ceph-mon[26293]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:06.518 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:06 smithi070 ceph-mon[21827]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:06.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:38:06] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-04-24T16:38:08.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:08 smithi156 ceph-mon[22679]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:08.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:08 smithi070 ceph-mon[26293]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:08.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:08 smithi070 ceph-mon[21827]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:09.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:09 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:09.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:09 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:09.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:09 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:10.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:10 smithi156 ceph-mon[22679]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:10.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:10 smithi070 ceph-mon[26293]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:10.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:10 smithi070 ceph-mon[21827]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:11.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:11 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:11.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:11 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:11.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:11 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:12.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:12 smithi156 ceph-mon[22679]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:12.443 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:12 smithi070 ceph-mon[26293]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:12.443 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:12 smithi070 ceph-mon[21827]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:14.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:13 smithi070 ceph-mon[21827]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:14.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:13 smithi070 ceph-mon[26293]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:14.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:13 smithi156 ceph-mon[22679]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:14.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:38:16.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:16 smithi070 ceph-mon[21827]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:16.343 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:16 smithi070 ceph-mon[26293]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:16.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:16 smithi156 ceph-mon[22679]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:16.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:38:16] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-04-24T16:38:16.792 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:38:16.792 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (9m) 2m ago 16m 24.7M - 0.23.0 ba2b418f427c 609b591dc19c 2024-04-24T16:38:16.792 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (8m) 2m ago 16m 56.2M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (2m) 2m ago 16m 48.0M - 3.5 e1d6a67b021e d600c1db7cc3 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283 running (11m) 2m ago 21m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 432b981bb958 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (6m) 2m ago 22m 499M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (22m) 2m ago 22m 63.0M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (21m) 2m ago 21m 48.5M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (21m) 2m ago 21m 48.5M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (9m) 2m ago 17m 16.3M - 1.3.1 1dbe0e931976 a0dadc7082e4 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (9m) 2m ago 17m 14.8M - 1.3.1 1dbe0e931976 cda127a7d141 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (21m) 2m ago 21m 55.5M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (20m) 2m ago 20m 56.4M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (20m) 2m ago 20m 50.2M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (19m) 2m ago 19m 50.6M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:38:16.793 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (19m) 2m ago 19m 52.2M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:38:16.794 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (19m) 2m ago 19m 51.7M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:38:16.794 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (18m) 2m ago 18m 49.4M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:38:16.794 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (18m) 2m ago 18m 54.0M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:38:16.794 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (2m) 2m ago 17m 41.4M - 2.33.4 514e6a882f6e 072ede669133 2024-04-24T16:38:16.794 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (16m) 2m ago 16m 73.2M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:38:16.794 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (16m) 2m ago 16m 73.6M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:38:17.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions' 2024-04-24T16:38:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:18 smithi070 ceph-mon[21827]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:38:18.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:18 smithi070 ceph-mon[21827]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:18.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:18 smithi070 ceph-mon[26293]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:38:18.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:18 smithi070 ceph-mon[26293]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:18.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:18 smithi156 ceph-mon[22679]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:38:18.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:18 smithi156 ceph-mon[22679]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:19.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:19 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:19.455 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:19 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:19.455 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:38:19.456 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:38:19.457 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:38:19.457 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:38:19.457 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:19 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:20.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph -s' 2024-04-24T16:38:20.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:20 smithi070 ceph-mon[26293]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:20.402 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:20 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1190209531' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:38:20.402 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:20 smithi070 ceph-mon[21827]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:20.402 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:20 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1190209531' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:38:20.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:20 smithi156 ceph-mon[22679]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:20.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:20 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1190209531' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:38:22.140 INFO:teuthology.orchestra.run.smithi070.stdout: cluster: 2024-04-24T16:38:22.140 INFO:teuthology.orchestra.run.smithi070.stdout: id: be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: health: HEALTH_OK 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: services: 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: mgr: y(active, since 3m), standbys: x 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: data: 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: pools: 6 pools, 161 pgs 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: objects: 209 objects, 585 KiB 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: usage: 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: pgs: 161 active+clean 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:38:22.141 INFO:teuthology.orchestra.run.smithi070.stdout: io: 2024-04-24T16:38:22.142 INFO:teuthology.orchestra.run.smithi070.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-24T16:38:22.142 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-24T16:38:22.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:22 smithi156 ceph-mon[22679]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:22.462 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:22 smithi070 ceph-mon[26293]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:22.462 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:22 smithi070 ceph-mon[21827]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:22.738 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:38:23.333 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:23 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3148223240' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:38:23.333 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:23 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3148223240' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:38:23.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:23 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3148223240' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T16:38:24.396 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:24 smithi070 ceph-mon[26293]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:24.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:24 smithi070 ceph-mon[21827]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:24.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:24 smithi156 ceph-mon[22679]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:25.038 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_OK 2024-04-24T16:38:25.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:25 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3087603577' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:38:25.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:25 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3087603577' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:38:25.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:25 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3087603577' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:38:25.652 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-24T16:38:26.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:26 smithi070 ceph-mon[26293]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:26.215 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:26 smithi070 ceph-mon[26293]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:26.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:26 smithi070 ceph-mon[21827]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:26.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:26 smithi070 ceph-mon[21827]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:26.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:26 smithi156 ceph-mon[22679]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:26.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:26 smithi156 ceph-mon[22679]: from='mgr.25120 172.21.15.70:0/1833938955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:26.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:38:26] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-04-24T16:38:27.808 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:38:28.409 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:28 smithi070 ceph-mon[26293]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:28.409 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:28 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2191329101' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:38:28.409 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:28 smithi070 ceph-mon[21827]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:28.409 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:28 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2191329101' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:38:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:28 smithi156 ceph-mon[22679]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:38:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:28 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2191329101' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:38:28.432 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph mgr fail' 2024-04-24T16:38:29.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:29 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:29.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:29 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:29.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:29 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:30.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:30 smithi070 ceph-mon[21827]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:30.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:30 smithi070 ceph-mon[26293]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:30.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:30 smithi156 ceph-mon[22679]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:38:31.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1462573869' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:38:31.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-mon[22679]: osdmap e104: 8 total, 8 up, 8 in 2024-04-24T16:38:31.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-mon[22679]: Standby manager daemon y started 2024-04-24T16:38:31.409 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:31] ENGINE Bus STOPPING 2024-04-24T16:38:31.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:38:31.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:31] ENGINE Bus STOPPED 2024-04-24T16:38:31.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:31] ENGINE Bus STARTING 2024-04-24T16:38:31.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:31] ENGINE Serving on http://:::9283 2024-04-24T16:38:31.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:31] ENGINE Bus STARTED 2024-04-24T16:38:31.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1462573869' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:38:31.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-mon[26293]: osdmap e104: 8 total, 8 up, 8 in 2024-04-24T16:38:31.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-mon[26293]: Standby manager daemon y started 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1462573869' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-mon[21827]: osdmap e104: 8 total, 8 up, 8 in 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-mon[21827]: Standby manager daemon y started 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:31.109+0000 7f8c84cec700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ignoring --setuser ceph since I am not root 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ignoring --setgroup ceph since I am not root 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-mgr[60576]: -- 172.21.15.70:0/808845026 <== mon.1 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562abecb85a0 con 0x562abec8c800 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:31.293+0000 7ff53208e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:38:31.485 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:31.485+0000 7ff53208e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:38:31.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:31.596+0000 7ff53208e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:38:31.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'sleep 180' 2024-04-24T16:38:32.101 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:31.927+0000 7ff53208e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:38:32.101 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:32.049+0000 7ff53208e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:38:32.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1462573869' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:38:32.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: mgrmap e33: x(active, starting, since 0.699825s), standbys: y 2024-04-24T16:38:32.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:38:32.387 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:38:32.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: Manager daemon x is now available 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: [24/Apr/2024:16:38:31] ENGINE Bus STARTING 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[26293]: [24/Apr/2024:16:38:32] ENGINE Serving on http://172.21.15.156:8765 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1462573869' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: mgrmap e33: x(active, starting, since 0.699825s), standbys: y 2024-04-24T16:38:32.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:38:32.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: Manager daemon x is now available 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: [24/Apr/2024:16:38:31] ENGINE Bus STARTING 2024-04-24T16:38:32.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:32 smithi070 ceph-mon[21827]: [24/Apr/2024:16:38:32] ENGINE Serving on http://172.21.15.156:8765 2024-04-24T16:38:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1462573869' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-24T16:38:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: mgrmap e33: x(active, starting, since 0.699825s), standbys: y 2024-04-24T16:38:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:38:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:38:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:38:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: Manager daemon x is now available 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: [24/Apr/2024:16:38:31] ENGINE Bus STARTING 2024-04-24T16:38:32.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:32 smithi156 ceph-mon[22679]: [24/Apr/2024:16:38:32] ENGINE Serving on http://172.21.15.156:8765 2024-04-24T16:38:33.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[26293]: mgrmap e34: x(active, since 1.70495s), standbys: y 2024-04-24T16:38:33.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[26293]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:33.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[26293]: [24/Apr/2024:16:38:32] ENGINE Serving on https://172.21.15.156:7150 2024-04-24T16:38:33.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[26293]: [24/Apr/2024:16:38:32] ENGINE Bus STARTED 2024-04-24T16:38:33.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[21827]: mgrmap e34: x(active, since 1.70495s), standbys: y 2024-04-24T16:38:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[21827]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[21827]: [24/Apr/2024:16:38:32] ENGINE Serving on https://172.21.15.156:7150 2024-04-24T16:38:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-mon[21827]: [24/Apr/2024:16:38:32] ENGINE Bus STARTED 2024-04-24T16:38:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:33 smithi156 ceph-mon[22679]: mgrmap e34: x(active, since 1.70495s), standbys: y 2024-04-24T16:38:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:33 smithi156 ceph-mon[22679]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:33 smithi156 ceph-mon[22679]: [24/Apr/2024:16:38:32] ENGINE Serving on https://172.21.15.156:7150 2024-04-24T16:38:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:33 smithi156 ceph-mon[22679]: [24/Apr/2024:16:38:32] ENGINE Bus STARTED 2024-04-24T16:38:33.673 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:33.308+0000 7ff53208e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:38:33.673 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:33.498+0000 7ff53208e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:38:33.673 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:33.673+0000 7ff53208e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:38:34.140 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:34 smithi156 ceph-mon[22679]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:34.140 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:34 smithi156 ceph-mon[22679]: mgrmap e35: x(active, since 2s), standbys: y 2024-04-24T16:38:34.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:33.981+0000 7ff53208e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:38:34.290 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:34 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:34.082+0000 7ff53208e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:38:34.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:34 smithi070 ceph-mon[26293]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:34.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:34 smithi070 ceph-mon[26293]: mgrmap e35: x(active, since 2s), standbys: y 2024-04-24T16:38:34.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:34 smithi070 ceph-mon[21827]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:34.291 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:34 smithi070 ceph-mon[21827]: mgrmap e35: x(active, since 2s), standbys: y 2024-04-24T16:38:34.542 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:34 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:34.290+0000 7ff53208e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:38:34.542 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:34 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:34.499+0000 7ff53208e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:38:35.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:35 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:35.130+0000 7ff53208e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:38:35.689 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:35 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:35.591+0000 7ff53208e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:38:36.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:35 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:35.690+0000 7ff53208e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:38:36.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:35 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:35.905+0000 7ff53208e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:38:36.374 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[26293]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:36.374 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[26293]: mgrmap e36: x(active, since 4s), standbys: y 2024-04-24T16:38:36.375 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:36.375 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:36.375 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[21827]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:36.375 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[21827]: mgrmap e36: x(active, since 4s), standbys: y 2024-04-24T16:38:36.375 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:36.375 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:36 smithi156 ceph-mon[22679]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:36 smithi156 ceph-mon[22679]: mgrmap e36: x(active, since 4s), standbys: y 2024-04-24T16:38:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:36 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:36 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:36.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:36.459+0000 7ff53208e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:38:36.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:36.554+0000 7ff53208e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:38:37.194 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:36.828+0000 7ff53208e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:38:37.194 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:36.970+0000 7ff53208e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:38:37.194 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:37.074+0000 7ff53208e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:38:37.195 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:37.194+0000 7ff53208e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:38:37.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:37.510+0000 7ff53208e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:38:38.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:37 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:37 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:37 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:37 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.166 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.166 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.166 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.166 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.166 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:37 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:38.508 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:38.166+0000 7ff53208e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:38:38.508 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:38.402+0000 7ff53208e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:38:38.508 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:38:38] ENGINE Bus STARTING 2024-04-24T16:38:38.509 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: CherryPy Checker: 2024-04-24T16:38:38.509 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: The Application mounted at '' has an empty config. 2024-04-24T16:38:38.509 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:38:38.509 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:38:38] ENGINE Serving on http://:::9283 2024-04-24T16:38:38.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:38:38] ENGINE Bus STARTED 2024-04-24T16:38:39.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:39.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:39.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: Standby manager daemon y restarted 2024-04-24T16:38:39.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: Standby manager daemon y started 2024-04-24T16:38:39.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:38:39.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:38:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:38:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:38:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:38 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:38:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: Standby manager daemon y restarted 2024-04-24T16:38:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: Standby manager daemon y started 2024-04-24T16:38:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:38:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:38:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='client.34982 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: Standby manager daemon y restarted 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: Standby manager daemon y started 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T16:38:39.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:38:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T16:38:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:38:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:38 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:38:40.132 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: mgrmap e37: x(active, since 8s), standbys: y 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:40.133 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: mgrmap e37: x(active, since 8s), standbys: y 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.134 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:40.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:40.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:39 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:40.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:38:40.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: mgrmap e37: x(active, since 8s), standbys: y 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:40.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:40.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:39 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:41.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[21827]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:38:41.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:38:41.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:38:41.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:38:41.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:38:41.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[26293]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:38:41.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:38:41.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:38:41.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:38:41.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:40 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:38:41.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:40 smithi156 ceph-mon[22679]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:38:41.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:40 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:38:41.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:40 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:38:41.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:40 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:38:41.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:40 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:38:43.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-mon[22679]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:38:43.159 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:38:42.814+0000 7fe984e54700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:43.159 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:43.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:43.161 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:43.161 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:43.161 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:43.161 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:43.161 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:43.161 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:38:43.161 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:43.162 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:38:43.163 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:43.164 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:42 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:43.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:42 smithi070 ceph-mon[21827]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:38:43.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:42 smithi070 ceph-mon[26293]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:38:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:43 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:38:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:43 smithi156 ceph-mon[22679]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:43 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:44.221 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:44.222 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:44.223 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:44.224 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:43 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.225 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:44.226 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:38:44.227 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.228 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:44.229 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:43 smithi070 ceph-mon[21827]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:44.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:43 smithi070 ceph-mon[26293]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:38:45.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:44 smithi156 ceph-mon[22679]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:38:45.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:44 smithi156 ceph-mon[22679]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:38:45.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:44 smithi070 ceph-mon[21827]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:38:45.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:44 smithi070 ceph-mon[21827]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:38:45.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:44 smithi070 ceph-mon[26293]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:38:45.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:44 smithi070 ceph-mon[26293]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-24T16:38:46.539 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 systemd[1]: Stopping Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:38:46.539 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 bash[54473]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:38:46.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:38:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:38:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.539Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-24T16:38:46.823 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-24T16:38:46.824 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-24T16:38:46.824 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[51522]: ts=2024-04-24T16:38:46.540Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-24T16:38:46.824 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 podman[54484]: 2024-04-24 16:38:46.550253426 +0000 UTC m=+0.118013225 container died 072ede669133e8e8520287378724307979c0b98476e8589d206d2f83c2eacb13 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:38:47.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-mon[22679]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:47.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:46 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:47.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 podman[54484]: 2024-04-24 16:38:46.923658744 +0000 UTC m=+0.491418542 container cleanup 072ede669133e8e8520287378724307979c0b98476e8589d206d2f83c2eacb13 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:38:47.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 bash[54484]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a 2024-04-24T16:38:47.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:46 smithi156 bash[54510]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:38:47.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:46 smithi070 ceph-mon[21827]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:47.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:46 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:47.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:46 smithi070 ceph-mon[26293]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:47.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:46 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:38:47.466 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 podman[54498]: 2024-04-24 16:38:47.216739297 +0000 UTC m=+0.669184697 container remove 072ede669133e8e8520287378724307979c0b98476e8589d206d2f83c2eacb13 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:38:47.466 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-24T16:38:47.466 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 systemd[1]: Stopped Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:38:47.466 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 1.385s CPU time. 2024-04-24T16:38:47.466 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 systemd[1]: Starting Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:38:47.466 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 podman[54575]: 2024-04-24 16:38:47.354229705 +0000 UTC m=+0.019668727 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-24T16:38:47.466 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 podman[54575]: 2024-04-24 16:38:47.467742531 +0000 UTC m=+0.133181538 container create 052495724fdc11593ca3a330b66fdb3ef467a6ced5e057cd2b0fe62aeac561d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:38:47.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 podman[54575]: 2024-04-24 16:38:47.67850008 +0000 UTC m=+0.343939099 container init 052495724fdc11593ca3a330b66fdb3ef467a6ced5e057cd2b0fe62aeac561d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:38:47.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 podman[54575]: 2024-04-24 16:38:47.681032641 +0000 UTC m=+0.346471649 container start 052495724fdc11593ca3a330b66fdb3ef467a6ced5e057cd2b0fe62aeac561d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:38:47.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.704Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-24T16:38:47.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.704Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-24T16:38:47.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.704Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-24T16:38:47.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.704Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi156 (none))" 2024-04-24T16:38:47.760 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.704Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.704Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.706Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.707Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.707Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.709Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.709Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.814µs 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.709Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-24T16:38:47.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 bash[54575]: 052495724fdc11593ca3a330b66fdb3ef467a6ced5e057cd2b0fe62aeac561d7 2024-04-24T16:38:48.141 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 systemd[1]: Started Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:38:48.141 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:47.909Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-04-24T16:38:48.142 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:47] ENGINE Bus STOPPING 2024-04-24T16:38:48.142 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:38:48.142 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:47] ENGINE Bus STOPPED 2024-04-24T16:38:48.142 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:47] ENGINE Bus STARTING 2024-04-24T16:38:48.142 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:47] ENGINE Serving on http://:::9283 2024-04-24T16:38:48.142 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:47 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:38:47] ENGINE Bus STARTED 2024-04-24T16:38:48.409 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.143Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-04-24T16:38:48.409 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.358Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.361Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.361Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.361Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=50.535µs wal_replay_duration=651.499824ms total_replay_duration=651.56383ms 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.365Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.365Z caller=main.go:947 level=info msg="TSDB started" 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.365Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.388Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.251656ms db_storage=771ns remote_storage=1.174µs web_handler=391ns query_engine=717ns scrape=386.619µs scrape_sd=94.591µs notify=14.667µs notify_sd=20.052µs rules=21.401049ms 2024-04-24T16:38:48.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:38:48.388Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-24T16:38:49.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:38:49.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:49.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:38:49.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:38:49.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:38:49.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156:9095"}]: dispatch 2024-04-24T16:38:49.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:49.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:38:51.074 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:50 smithi156 ceph-mon[22679]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:51.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:50 smithi070 ceph-mon[21827]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:51.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:50 smithi070 ceph-mon[26293]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T16:38:53.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:38:53.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:53.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:38:53.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:53.139 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:53.140 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:53.140 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:38:53.140 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.140 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.140 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:53.140 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:38:53.140 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.141 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:53.141 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:53.141 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:52 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:53.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:38:53.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:53.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:38:53.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:38:53.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:53.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:38:53.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:52 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:38:53.958 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:53 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:38:53.958 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:53 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:38:53.958 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:53 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:38:53.958 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:53 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:38:54.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:53 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:38:54.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:53 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:38:55.016 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:54 smithi070 ceph-mon[26293]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:38:55.016 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:54 smithi070 ceph-mon[21827]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:38:55.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:54 smithi156 ceph-mon[22679]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-24T16:38:55.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:38:55.654+0000 7fe984e54700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:55.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:55.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:55.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:55.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:55.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:55.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:55.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:55.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:55.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:55.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:55.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:38:55.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:38:55.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:38:55.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:55.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:56.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:38:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:38:56] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:56 smithi156 ceph-mon[22679]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:56 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:57.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:57.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:56 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2023610107' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:38:57.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:56 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1491514870' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3708709489"}]: dispatch 2024-04-24T16:38:57.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[21827]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:57.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.292 INFO:journalctl@ceph.mon.a.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.292 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.292 INFO:journalctl@ceph.mon.a.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:57.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2023610107' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1491514870' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3708709489"}]: dispatch 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[26293]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:38:57.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:57.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:38:57.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi-foo-smithi070-gcoqkr 2024-04-24T16:38:57.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: /usr/bin/podman: stderr Error: no such container ceph-be31be9a-0255-11ef-bc93-c7b262605968-iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:38:57.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2023610107' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:38:57.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:56 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1491514870' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3708709489"}]: dispatch 2024-04-24T16:38:58.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:57 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1491514870' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3708709489"}]': finished 2024-04-24T16:38:58.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:57 smithi156 ceph-mon[22679]: osdmap e105: 8 total, 8 up, 8 in 2024-04-24T16:38:58.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:57 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1184333421' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3264881507"}]: dispatch 2024-04-24T16:38:58.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:57 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1491514870' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3708709489"}]': finished 2024-04-24T16:38:58.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:57 smithi070 ceph-mon[21827]: osdmap e105: 8 total, 8 up, 8 in 2024-04-24T16:38:58.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:57 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1184333421' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3264881507"}]: dispatch 2024-04-24T16:38:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:57 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1491514870' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3708709489"}]': finished 2024-04-24T16:38:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:57 smithi070 ceph-mon[26293]: osdmap e105: 8 total, 8 up, 8 in 2024-04-24T16:38:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:57 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1184333421' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3264881507"}]: dispatch 2024-04-24T16:38:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:58 smithi156 ceph-mon[22679]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-24T16:38:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:58 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1184333421' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3264881507"}]': finished 2024-04-24T16:38:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:58 smithi156 ceph-mon[22679]: osdmap e106: 8 total, 8 up, 8 in 2024-04-24T16:38:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:58 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4252445959' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3230468899"}]: dispatch 2024-04-24T16:38:59.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[26293]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-24T16:38:59.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1184333421' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3264881507"}]': finished 2024-04-24T16:38:59.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[26293]: osdmap e106: 8 total, 8 up, 8 in 2024-04-24T16:38:59.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4252445959' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3230468899"}]: dispatch 2024-04-24T16:38:59.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[21827]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-24T16:38:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1184333421' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3264881507"}]': finished 2024-04-24T16:38:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[21827]: osdmap e106: 8 total, 8 up, 8 in 2024-04-24T16:38:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:58 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4252445959' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3230468899"}]: dispatch 2024-04-24T16:39:00.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:59 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4252445959' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3230468899"}]': finished 2024-04-24T16:39:00.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:59 smithi156 ceph-mon[22679]: osdmap e107: 8 total, 8 up, 8 in 2024-04-24T16:39:00.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:59 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/3323918927' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]: dispatch 2024-04-24T16:39:00.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:38:59 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]: dispatch 2024-04-24T16:39:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4252445959' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3230468899"}]': finished 2024-04-24T16:39:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[26293]: osdmap e107: 8 total, 8 up, 8 in 2024-04-24T16:39:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/3323918927' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]: dispatch 2024-04-24T16:39:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]: dispatch 2024-04-24T16:39:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4252445959' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/3230468899"}]': finished 2024-04-24T16:39:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[21827]: osdmap e107: 8 total, 8 up, 8 in 2024-04-24T16:39:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/3323918927' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]: dispatch 2024-04-24T16:39:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:38:59 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]: dispatch 2024-04-24T16:39:01.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:00 smithi156 ceph-mon[22679]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:39:01.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:00 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]': finished 2024-04-24T16:39:01.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:00 smithi156 ceph-mon[22679]: osdmap e108: 8 total, 8 up, 8 in 2024-04-24T16:39:01.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:00 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2507002280' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]: dispatch 2024-04-24T16:39:01.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:00 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]: dispatch 2024-04-24T16:39:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[26293]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:39:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]': finished 2024-04-24T16:39:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[26293]: osdmap e108: 8 total, 8 up, 8 in 2024-04-24T16:39:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2507002280' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]: dispatch 2024-04-24T16:39:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]: dispatch 2024-04-24T16:39:01.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[21827]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:39:01.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1909679688"}]': finished 2024-04-24T16:39:01.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[21827]: osdmap e108: 8 total, 8 up, 8 in 2024-04-24T16:39:01.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2507002280' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]: dispatch 2024-04-24T16:39:01.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:00 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]: dispatch 2024-04-24T16:39:02.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:01 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]': finished 2024-04-24T16:39:02.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:01 smithi156 ceph-mon[22679]: osdmap e109: 8 total, 8 up, 8 in 2024-04-24T16:39:02.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:01 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:02.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:01 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4179952557' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]: dispatch 2024-04-24T16:39:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:01 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]: dispatch 2024-04-24T16:39:02.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]': finished 2024-04-24T16:39:02.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[26293]: osdmap e109: 8 total, 8 up, 8 in 2024-04-24T16:39:02.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:02.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4179952557' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]: dispatch 2024-04-24T16:39:02.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]: dispatch 2024-04-24T16:39:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2763885013"}]': finished 2024-04-24T16:39:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[21827]: osdmap e109: 8 total, 8 up, 8 in 2024-04-24T16:39:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4179952557' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]: dispatch 2024-04-24T16:39:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:01 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]: dispatch 2024-04-24T16:39:03.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:02 smithi156 ceph-mon[22679]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:39:03.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:02 smithi156 ceph-mon[22679]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]': finished 2024-04-24T16:39:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:02 smithi156 ceph-mon[22679]: osdmap e110: 8 total, 8 up, 8 in 2024-04-24T16:39:03.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:02 smithi070 ceph-mon[26293]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:39:03.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:02 smithi070 ceph-mon[26293]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]': finished 2024-04-24T16:39:03.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:02 smithi070 ceph-mon[26293]: osdmap e110: 8 total, 8 up, 8 in 2024-04-24T16:39:03.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:02 smithi070 ceph-mon[21827]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:39:03.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:02 smithi070 ceph-mon[21827]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1909679688"}]': finished 2024-04-24T16:39:03.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:02 smithi070 ceph-mon[21827]: osdmap e110: 8 total, 8 up, 8 in 2024-04-24T16:39:05.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:04 smithi156 ceph-mon[22679]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:05.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:04 smithi070 ceph-mon[26293]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:05.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:04 smithi070 ceph-mon[21827]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:06.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:39:06] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-04-24T16:39:07.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:06 smithi156 ceph-mon[22679]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:07.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:06 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:07.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:06 smithi070 ceph-mon[26293]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:07.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:06 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:07.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:06 smithi070 ceph-mon[21827]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:06 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:09.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:08 smithi156 ceph-mon[22679]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-24T16:39:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:08 smithi070 ceph-mon[26293]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-24T16:39:09.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:08 smithi070 ceph-mon[21827]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-24T16:39:11.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:10 smithi156 ceph-mon[22679]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:11.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:10 smithi070 ceph-mon[26293]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:11.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:10 smithi070 ceph-mon[21827]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:13.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:12 smithi156 ceph-mon[22679]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:39:13.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:12 smithi070 ceph-mon[26293]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:39:13.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:12 smithi070 ceph-mon[21827]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:39:15.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:14 smithi156 ceph-mon[22679]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:39:15.241 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:14 smithi070 ceph-mon[26293]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:39:15.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:14 smithi070 ceph-mon[21827]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:39:16.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:39:16] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-04-24T16:39:17.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:16 smithi156 ceph-mon[22679]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:17.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:16 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:17.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:16 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:17.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:16 smithi070 ceph-mon[26293]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:17.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:16 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:17.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:16 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:16 smithi070 ceph-mon[21827]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:16 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:17.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:16 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:19.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:18 smithi156 ceph-mon[22679]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:18 smithi070 ceph-mon[26293]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:18 smithi070 ceph-mon[21827]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:21.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:20 smithi156 ceph-mon[22679]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:21.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:20 smithi070 ceph-mon[26293]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:21.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:20 smithi070 ceph-mon[21827]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:23.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:22 smithi156 ceph-mon[22679]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:23.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:22 smithi070 ceph-mon[21827]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:23.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:22 smithi070 ceph-mon[26293]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:25.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:24 smithi070 ceph-mon[26293]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:25.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:24 smithi070 ceph-mon[21827]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:25.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:24 smithi156 ceph-mon[22679]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:26.609 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:39:26] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-04-24T16:39:27.181 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:26 smithi156 ceph-mon[22679]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:27.181 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:26 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:27.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:26 smithi070 ceph-mon[26293]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:27.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:26 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:27.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:26 smithi070 ceph-mon[21827]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:27.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:26 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:29.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:28 smithi070 ceph-mon[21827]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:29.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:28 smithi070 ceph-mon[26293]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:29.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:28 smithi156 ceph-mon[22679]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:31.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:30 smithi070 ceph-mon[21827]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:31.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:30 smithi070 ceph-mon[26293]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:31.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:30 smithi156 ceph-mon[22679]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:32.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:31 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:32.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:31 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:31 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:33.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:32 smithi070 ceph-mon[26293]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:33.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:32 smithi070 ceph-mon[21827]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:32 smithi156 ceph-mon[22679]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:34 smithi070 ceph-mon[21827]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:34 smithi070 ceph-mon[26293]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:34 smithi156 ceph-mon[22679]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:36.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:39:36] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-04-24T16:39:37.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:36 smithi070 ceph-mon[26293]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:39:37.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:36 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:37.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:36 smithi070 ceph-mon[21827]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:39:37.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:36 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:37.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:36 smithi156 ceph-mon[22679]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:39:37.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:36 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:39.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:38 smithi070 ceph-mon[26293]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:39.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:38 smithi070 ceph-mon[21827]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:39.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:38 smithi156 ceph-mon[22679]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:41.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:40 smithi070 ceph-mon[26293]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:41.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:40 smithi070 ceph-mon[21827]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:41.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:40 smithi156 ceph-mon[22679]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:43.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:42 smithi070 ceph-mon[26293]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:43.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:42 smithi070 ceph-mon[21827]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:43.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:42 smithi156 ceph-mon[22679]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:45.212 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:44 smithi156 ceph-mon[22679]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:45.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:44 smithi070 ceph-mon[26293]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:45.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:44 smithi070 ceph-mon[21827]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:46.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:39:46] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-04-24T16:39:47.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:46 smithi070 ceph-mon[21827]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:47.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:46 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:47.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:46 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:47.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:46 smithi070 ceph-mon[26293]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:47.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:46 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:47.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:46 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:47.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:46 smithi156 ceph-mon[22679]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:46 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:46 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:39:49.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:48 smithi070 ceph-mon[21827]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:49.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:48 smithi070 ceph-mon[26293]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:49.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:48 smithi156 ceph-mon[22679]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:51.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:50 smithi070 ceph-mon[21827]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:51.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:50 smithi070 ceph-mon[26293]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:51.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:50 smithi156 ceph-mon[22679]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:53.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:52 smithi070 ceph-mon[21827]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:53.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:52 smithi070 ceph-mon[26293]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:53.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:52 smithi156 ceph-mon[22679]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:55.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:54 smithi070 ceph-mon[21827]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:55.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:54 smithi070 ceph-mon[26293]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:55.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:54 smithi156 ceph-mon[22679]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:56.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:55 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:39:56.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:55 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:39:56.330 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:55 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:39:56.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:39:56] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-04-24T16:39:57.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:57.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:57.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:39:57.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:39:57.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:39:57.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:39:57.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:39:57.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:39:57.208 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:56 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:39:57.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:39:57.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:39:57.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:39:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:39:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:39:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:39:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:39:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:56 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:39:58.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:57 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:39:58.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:57 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:39:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:57 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:39:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:57 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:39:58.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:57 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:39:58.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:57 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:39:58.126+0000 7fe984e54700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:39:58.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:39:58.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:39:58.412 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:39:58.413 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:39:58.414 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:39:58.415 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:39:58.415 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:39:58.415 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:39:58.415 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:39:59.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:58 smithi070 ceph-mon[21827]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:59.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:58 smithi070 ceph-mon[26293]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:39:59.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:58 smithi156 ceph-mon[22679]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:39:59 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:40:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:40:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:40:00.295 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:40:00.296 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:39:59 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.298 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:40:00.299 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:40:00.300 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:39:59 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:40:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:40:00.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:40:00.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:40:00.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:40:00.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:40:00.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:40:00.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:40:00.415 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:40:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:00 smithi070 ceph-mon[26293]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:00 smithi070 ceph-mon[26293]: overall HEALTH_OK 2024-04-24T16:40:01.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:00 smithi070 ceph-mon[21827]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:01.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:00 smithi070 ceph-mon[21827]: overall HEALTH_OK 2024-04-24T16:40:01.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:00 smithi156 ceph-mon[22679]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:01.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:00 smithi156 ceph-mon[22679]: overall HEALTH_OK 2024-04-24T16:40:02.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:01 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:01 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:02.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:01 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:03.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:02 smithi070 ceph-mon[21827]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:03.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:02 smithi070 ceph-mon[26293]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:03.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:02 smithi156 ceph-mon[22679]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:05.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:04 smithi070 ceph-mon[21827]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:05.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:04 smithi070 ceph-mon[26293]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:05.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:04 smithi156 ceph-mon[22679]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:06.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:40:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:40:06] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-04-24T16:40:07.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:06 smithi070 ceph-mon[21827]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:07.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:06 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:06 smithi070 ceph-mon[26293]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:06 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:07.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:06 smithi156 ceph-mon[22679]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:07.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:06 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:08 smithi070 ceph-mon[21827]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:08 smithi070 ceph-mon[26293]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:09.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:08 smithi156 ceph-mon[22679]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:11.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:10 smithi070 ceph-mon[21827]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:11.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:10 smithi070 ceph-mon[26293]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:11.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:10 smithi156 ceph-mon[22679]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:13.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:12 smithi070 ceph-mon[21827]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:13.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:12 smithi070 ceph-mon[26293]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:13.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:12 smithi156 ceph-mon[22679]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:15.251 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:14 smithi070 ceph-mon[26293]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:15.251 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:14 smithi070 ceph-mon[21827]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:15.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:14 smithi156 ceph-mon[22679]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:16.616 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:40:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:40:16] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-04-24T16:40:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:17 smithi070 ceph-mon[21827]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:17 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:17 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:17 smithi070 ceph-mon[26293]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:17 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:17 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:17.363 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:17 smithi156 ceph-mon[22679]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:17.363 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:17 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:17.363 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:17 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:19 smithi070 ceph-mon[21827]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:19 smithi070 ceph-mon[26293]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:19.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:19 smithi156 ceph-mon[22679]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:21.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:21 smithi070 ceph-mon[21827]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:21.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:21 smithi070 ceph-mon[26293]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:21.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:21 smithi156 ceph-mon[22679]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:23.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:23 smithi070 ceph-mon[21827]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:23.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:23 smithi070 ceph-mon[26293]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:23.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:23 smithi156 ceph-mon[22679]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:25.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:25 smithi070 ceph-mon[21827]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:25.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:25 smithi070 ceph-mon[26293]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:25.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:25 smithi156 ceph-mon[22679]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:26.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:40:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:40:26] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-04-24T16:40:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:27 smithi156 ceph-mon[22679]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:27.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:27 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:27.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:27 smithi070 ceph-mon[26293]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:27.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:27 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:27.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:27 smithi070 ceph-mon[21827]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:27.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:27 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:29.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:29 smithi070 ceph-mon[21827]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:29.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:29 smithi070 ceph-mon[26293]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:29.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:29 smithi156 ceph-mon[22679]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:31.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:31 smithi070 ceph-mon[21827]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:31.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:31 smithi070 ceph-mon[26293]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:31.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:31 smithi156 ceph-mon[22679]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:32.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:32 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:32.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:32 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:32 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:33.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:33 smithi070 ceph-mon[26293]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:33.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:33 smithi070 ceph-mon[21827]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:33 smithi156 ceph-mon[22679]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:35 smithi070 ceph-mon[26293]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:35 smithi070 ceph-mon[21827]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:35 smithi156 ceph-mon[22679]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:36.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:40:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:40:36] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-04-24T16:40:37.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:37 smithi070 ceph-mon[26293]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:37.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:37 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:37.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:37 smithi070 ceph-mon[21827]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:37 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:37.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:37 smithi156 ceph-mon[22679]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:37.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:37 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:39.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:39 smithi070 ceph-mon[26293]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:39 smithi070 ceph-mon[21827]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:39.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:39 smithi156 ceph-mon[22679]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:41.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:41 smithi070 ceph-mon[26293]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:41.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:41 smithi070 ceph-mon[21827]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:41.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:41 smithi156 ceph-mon[22679]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:43.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:43 smithi070 ceph-mon[26293]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:43.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:43 smithi070 ceph-mon[21827]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:43.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:43 smithi156 ceph-mon[22679]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:45.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:45 smithi070 ceph-mon[26293]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:45.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:45 smithi070 ceph-mon[21827]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:45.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:45 smithi156 ceph-mon[22679]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:46.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:40:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:40:46] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-04-24T16:40:47.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:47 smithi070 ceph-mon[26293]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:47.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:47 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:47.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:47 smithi070 ceph-mon[21827]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:47.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:47 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:47.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:47 smithi156 ceph-mon[22679]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:40:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:47 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:40:48.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:48 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:48.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:48 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:48.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:48 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:49.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:49 smithi156 ceph-mon[22679]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:49.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:49 smithi070 ceph-mon[26293]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:49.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:49 smithi070 ceph-mon[21827]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:51 smithi156 ceph-mon[22679]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:51.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:51 smithi070 ceph-mon[26293]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:51.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:51 smithi070 ceph-mon[21827]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:53.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:53 smithi156 ceph-mon[22679]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:53.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:53 smithi070 ceph-mon[26293]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:53.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:53 smithi070 ceph-mon[21827]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:55.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:55 smithi156 ceph-mon[22679]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:55.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:55 smithi070 ceph-mon[21827]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:55.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:55 smithi070 ceph-mon[26293]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:56.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:56 smithi156 ceph-mon[22679]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:56.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:56 smithi070 ceph-mon[26293]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:56.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:56 smithi070 ceph-mon[21827]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:40:56.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:40:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:40:56] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-04-24T16:40:57.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:57 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:57.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:57 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:57.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:57 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:40:59.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:40:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:40:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:40:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:40:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:40:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:40:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:58 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:40:59.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:59.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:40:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:40:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:40:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:40:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:40:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:40:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:58 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:41:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:59 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:41:00.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:40:59 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:41:00.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:59 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:41:00.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:40:59 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:41:00.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:59 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:41:00.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:40:59 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:41:00.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-mon[22679]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:00.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:00.643+0000 7fe984e54700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:00.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:41:00.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:00.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:00.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:00.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:00.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:00.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:00.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:00.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:41:00.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:00.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:00.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:00.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:01.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:00 smithi070 ceph-mon[21827]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:01.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:00 smithi070 ceph-mon[26293]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:01 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:02.161 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:02.162 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:02.163 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:02.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:02.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:02.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:02.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:02.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.164 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:01 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:02.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:01 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:02.293 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:02.295 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:02.296 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:01 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.297 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.298 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:02.299 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:02.300 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:01 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:02.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:02.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:02.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:02.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:02.302 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:02.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:01 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:03.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:02 smithi156 ceph-mon[22679]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:03.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:02 smithi070 ceph-mon[26293]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:03.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:02 smithi070 ceph-mon[21827]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:05.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:04 smithi156 ceph-mon[22679]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:05.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:04 smithi070 ceph-mon[26293]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:04 smithi070 ceph-mon[21827]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:06.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:41:06] "GET /metrics HTTP/1.1" 200 37913 "" "Prometheus/2.33.4" 2024-04-24T16:41:07.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:06 smithi156 ceph-mon[22679]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:07.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:06 smithi070 ceph-mon[26293]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:07.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:06 smithi070 ceph-mon[21827]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:08.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:07 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:08.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:07 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:08.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:07 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:09.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:08 smithi156 ceph-mon[22679]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:08 smithi070 ceph-mon[26293]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:09.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:08 smithi070 ceph-mon[21827]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:11.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:10 smithi156 ceph-mon[22679]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:11.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:10 smithi070 ceph-mon[26293]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:11.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:10 smithi070 ceph-mon[21827]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:13.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:12 smithi156 ceph-mon[22679]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:13.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:12 smithi070 ceph-mon[26293]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:13.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:12 smithi070 ceph-mon[21827]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:15.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:14 smithi156 ceph-mon[22679]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:15.241 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:14 smithi070 ceph-mon[26293]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:15.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:14 smithi070 ceph-mon[21827]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:16.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:41:16] "GET /metrics HTTP/1.1" 200 37915 "" "Prometheus/2.33.4" 2024-04-24T16:41:17.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:16 smithi156 ceph-mon[22679]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:17.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:16 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:16 smithi070 ceph-mon[21827]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:16 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:16 smithi070 ceph-mon[26293]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:16 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:18.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:17 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:18.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:17 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:18.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:17 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:18 smithi070 ceph-mon[26293]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:19.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:18 smithi070 ceph-mon[21827]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:19.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:18 smithi156 ceph-mon[22679]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:21.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:20 smithi070 ceph-mon[21827]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:21.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:20 smithi070 ceph-mon[26293]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:21.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:20 smithi156 ceph-mon[22679]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:23.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:22 smithi070 ceph-mon[21827]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:23.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:22 smithi070 ceph-mon[26293]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:23.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:22 smithi156 ceph-mon[22679]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:25.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:24 smithi070 ceph-mon[26293]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:25.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:24 smithi070 ceph-mon[21827]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:25.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:24 smithi156 ceph-mon[22679]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:26.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:26 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:41:26] "GET /metrics HTTP/1.1" 200 37915 "" "Prometheus/2.33.4" 2024-04-24T16:41:27.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:26 smithi070 ceph-mon[21827]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:27.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:26 smithi070 ceph-mon[26293]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:27.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:26 smithi156 ceph-mon[22679]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:28.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:27 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:28.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:27 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:28.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:27 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:29.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:28 smithi070 ceph-mon[21827]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:29.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:28 smithi070 ceph-mon[26293]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:29.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:28 smithi156 ceph-mon[22679]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:31.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:30 smithi070 ceph-mon[21827]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:31.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:30 smithi070 ceph-mon[26293]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:31.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:30 smithi156 ceph-mon[22679]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:32.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:31 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:32.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:31 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:32.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:31 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:33.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:32 smithi070 ceph-mon[21827]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:33.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:32 smithi070 ceph-mon[26293]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:33.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:32 smithi156 ceph-mon[22679]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:35.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:34 smithi070 ceph-mon[26293]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:35.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:34 smithi070 ceph-mon[21827]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:35.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:34 smithi156 ceph-mon[22679]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:35.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:41:36.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:36 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:41:36] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-04-24T16:41:37.017 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:36 smithi070 ceph-mon[26293]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:37.018 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:36 smithi070 ceph-mon[21827]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:37.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:36 smithi156 ceph-mon[22679]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:37.710 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (12m) 3m ago 20m 24.0M - 0.23.0 ba2b418f427c 609b591dc19c 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (12m) 2m ago 19m 56.4M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (6m) 3m ago 19m 51.1M - 3.5 e1d6a67b021e d600c1db7cc3 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283 running (15m) 2m ago 24m 496M - 19.0.0-3285-g5904d294 8bb2087275ff 432b981bb958 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (9m) 3m ago 26m 73.8M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (26m) 3m ago 26m 69.2M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (24m) 2m ago 24m 55.8M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (25m) 3m ago 25m 51.9M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (12m) 3m ago 20m 16.5M - 1.3.1 1dbe0e931976 a0dadc7082e4 2024-04-24T16:41:37.711 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (12m) 2m ago 20m 15.1M - 1.3.1 1dbe0e931976 cda127a7d141 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (24m) 3m ago 24m 55.6M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (23m) 3m ago 23m 56.4M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (23m) 3m ago 23m 50.2M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (23m) 3m ago 23m 50.7M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (22m) 2m ago 22m 52.4M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (22m) 2m ago 22m 51.7M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (22m) 2m ago 22m 49.9M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (21m) 2m ago 21m 54.3M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (2m) 2m ago 21m 44.1M - 2.33.4 514e6a882f6e 052495724fdc 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (19m) 3m ago 19m 73.8M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:41:37.712 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (19m) 2m ago 19m 74.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:41:38.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:37 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:38.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:37 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:38.340 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ls' 2024-04-24T16:41:38.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:37 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:39.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:38 smithi070 ceph-mon[26293]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:39.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:38 smithi070 ceph-mon[26293]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:38 smithi070 ceph-mon[21827]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:38 smithi070 ceph-mon[21827]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:39.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:38 smithi156 ceph-mon[22679]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:39.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:38 smithi156 ceph-mon[22679]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:40.540 INFO:teuthology.orchestra.run.smithi070.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager ?:9093,9094 1/1 3m ago 21m smithi070=a;count:1 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:grafana ?:3000 1/1 2m ago 21m smithi156=a;count:1 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo ?:5000 1/1 3m ago 19m count:1 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:mgr 2/2 3m ago 24m smithi070=y;smithi156=x;count:2 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:mon 3/3 3m ago 25m smithi070:172.21.15.70=a;smithi070:[v2:172.21.15.70:3301,v1:172.21.15.70:6790]=c;smithi156:172.21.15.156=b;count:3 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter ?:9100 2/2 3m ago 21m smithi070=a;smithi156=b;count:2 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:osd 8 3m ago - 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus ?:9095 1/1 2m ago 21m smithi156=a;count:1 2024-04-24T16:41:40.541 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo ?:8000 2/2 3m ago 19m count:2 2024-04-24T16:41:41.182 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:40 smithi070 ceph-mon[26293]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:41.183 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:40 smithi070 ceph-mon[21827]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:41.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions' 2024-04-24T16:41:41.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:40 smithi156 ceph-mon[22679]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:42.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:41 smithi070 ceph-mon[26293]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:42.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:41 smithi070 ceph-mon[21827]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:42.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:41 smithi156 ceph-mon[22679]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:43.202 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:41:43.203 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:41:43.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:43.204 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:41:43.204 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:41:43.204 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:41:43.204 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:41:43.204 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:41:43.218 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:42 smithi070 ceph-mon[26293]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:43.218 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:42 smithi070 ceph-mon[21827]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:42 smithi156 ceph-mon[22679]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:43.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-24T16:41:44.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:43 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2841265363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:41:44.241 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:43 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2841265363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:41:44.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:43 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2841265363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:41:45.233 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:44 smithi070 ceph-mon[26293]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:45.233 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:44 smithi070 ceph-mon[21827]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:45.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:44 smithi156 ceph-mon[22679]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:46.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:41:46] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-04-24T16:41:47.184 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:46 smithi070 ceph-mon[26293]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:47.184 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:46 smithi070 ceph-mon[26293]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:47.184 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:46 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:47.184 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:46 smithi070 ceph-mon[21827]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:47.184 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:46 smithi070 ceph-mon[21827]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:47.184 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:46 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:47.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:46 smithi156 ceph-mon[22679]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:46 smithi156 ceph-mon[22679]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:46 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:48.134 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:41:48.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:47 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:48.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:47 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:48.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:47 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:48.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-24T16:41:49.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:49.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:49.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:41:49.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:41:49.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:49.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:41:49.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:41:49.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:48 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:41:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:48 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:41:50.271 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:49 smithi070 ceph-mon[26293]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:41:50.271 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:49 smithi070 ceph-mon[26293]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:41:50.271 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:49 smithi070 ceph-mon[26293]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:41:50.272 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:49 smithi070 ceph-mon[21827]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:41:50.272 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:49 smithi070 ceph-mon[21827]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:41:50.272 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:49 smithi070 ceph-mon[21827]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:41:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:49 smithi156 ceph-mon[22679]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:41:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:49 smithi156 ceph-mon[22679]: Reconfiguring iscsi.foo.smithi070.gcoqkr (dependencies changed)... 2024-04-24T16:41:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:49 smithi156 ceph-mon[22679]: Reconfiguring daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:41:51.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:50 smithi070 ceph-mon[26293]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:51.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:50 smithi070 ceph-mon[21827]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:51.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:50 smithi156 ceph-mon[22679]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:51.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:51.551+0000 7fe984e54700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:51.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:41:51.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:51.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:51.911 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:51.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:51.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:51.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:51.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:51.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:41:51.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:51.912 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return future.result(timeout) 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return self.__get_result() 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: raise self._exception 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: use_current_daemon_image=reconfig, 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:51.913 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Creating ceph-iscsi config... 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: Traceback (most recent call last): 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: return _run_code(code, main_globals, None, 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: exec(code, run_globals) 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:51.914 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:51.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:51.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:51.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:51.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:51.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:51.915 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:51.984 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:41:52.669 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:41:52.669 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (12m) 3m ago 20m 24.0M - 0.23.0 ba2b418f427c 609b591dc19c 2024-04-24T16:41:52.669 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (12m) 3m ago 20m 56.4M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 running (6m) 3m ago 19m 51.1M - 3.5 e1d6a67b021e d600c1db7cc3 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283 running (15m) 3m ago 24m 496M - 19.0.0-3285-g5904d294 8bb2087275ff 432b981bb958 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (9m) 3m ago 26m 73.8M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (26m) 3m ago 26m 69.2M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (25m) 3m ago 25m 55.8M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (25m) 3m ago 25m 51.9M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (12m) 3m ago 21m 16.5M - 1.3.1 1dbe0e931976 a0dadc7082e4 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (12m) 3m ago 21m 15.1M - 1.3.1 1dbe0e931976 cda127a7d141 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (24m) 3m ago 24m 55.6M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (24m) 3m ago 24m 56.4M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (23m) 3m ago 23m 50.2M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (23m) 3m ago 23m 50.7M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (23m) 3m ago 23m 52.4M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:41:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (22m) 3m ago 22m 51.7M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:41:52.671 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (22m) 3m ago 22m 49.9M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:41:52.671 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (21m) 3m ago 21m 54.3M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:41:52.671 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (3m) 3m ago 21m 44.1M - 2.33.4 514e6a882f6e 052495724fdc 2024-04-24T16:41:52.671 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (19m) 3m ago 19m 73.8M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:41:52.671 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (19m) 3m ago 19m 74.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[26293]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[26293]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.044 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: return future.result(timeout) 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:53.045 INFO:journalctl@ceph.mon.c.smithi070.stdout: return self.__get_result() 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: raise self._exception 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:53.046 INFO:journalctl@ceph.mon.c.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:53.047 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[21827]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[21827]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:53.048 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:53.049 INFO:journalctl@ceph.mon.a.smithi070.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: return future.result(timeout) 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: return self.__get_result() 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: raise self._exception 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:53.050 INFO:journalctl@ceph.mon.a.smithi070.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:53.051 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.c.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.c.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.c.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[26293]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[26293]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:53.052 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:53.053 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:53.053 INFO:journalctl@ceph.mon.a.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:53.053 INFO:journalctl@ceph.mon.a.smithi070.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.053 INFO:journalctl@ceph.mon.a.smithi070.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.053 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[21827]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:41:53.053 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:52 smithi070 ceph-mon[21827]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:53.083 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:41:53.084 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:41:53.085 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:41:53.085 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:41:53.085 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:41:53.085 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:41:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:52 smithi156 ceph-mon[22679]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:41:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:52 smithi156 ceph-mon[22679]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1130, in _check_daemons 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2463, in _daemon_action 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 784, in wait_async 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: return future.result(timeout) 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: return self.__get_result() 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: raise self._exception 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1393, in _create_daemon 2024-04-24T16:41:53.412 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_current_daemon_image=reconfig, 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1710, in _run_cephadm 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: Creating ceph-iscsi config... 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: Non-zero exit code 1 from systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: Traceback (most recent call last): 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: exec(code, run_globals) 2024-04-24T16:41:53.413 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1123, in deploy_daemon 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: RuntimeError: Failed command: systemctl restart ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout: See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:52 smithi156 ceph-mon[22679]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:41:53.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:52 smithi156 ceph-mon[22679]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:53.426 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:41:53.426 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", 2024-04-24T16:41:53.426 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:41:53.426 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-24T16:41:53.426 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:41:53.427 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "", 2024-04-24T16:41:53.427 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc image", 2024-04-24T16:41:53.427 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:41:53.427 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:41:54.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4290937665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[21827]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4290937665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:41:54.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:54.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:54.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-24T16:41:54.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-24T16:41:54.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:53 smithi070 ceph-mon[26293]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/4290937665' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: from='mgr.34931 172.21.15.156:0/3701158790' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-24T16:41:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:53 smithi156 ceph-mon[22679]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:41:54.988 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:54.686+0000 7fe9f301d700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-24T16:41:54.988 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ignoring --setuser ceph since I am not root 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: ignoring --setgroup ceph since I am not root 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mgr[42890]: -- 172.21.15.156:0/2031617641 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fd804185a0 con 0x55fd803ecc00 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:54.865+0000 7ffadc613200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:41:54] ENGINE Bus STOPPING 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:41:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:41:54] ENGINE Bus STOPPED 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:41:54] ENGINE Bus STARTING 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:41:54] ENGINE Serving on http://:::9283 2024-04-24T16:41:54.989 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:41:54] ENGINE Bus STARTED 2024-04-24T16:41:55.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: Failing over to other MGR 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.34931 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-24T16:41:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: mgrmap e38: y(active, starting, since 0.996887s), standbys: x 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:41:55.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: Manager daemon y is now available 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:41:55.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: Failing over to other MGR 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.34931 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: mgrmap e38: y(active, starting, since 0.996887s), standbys: x 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:41:55.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: Manager daemon y is now available 2024-04-24T16:41:55.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:41:55.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:41:55.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:41:55.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:54 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:41:55.409 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:55.054+0000 7ffadc613200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:41:55.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:55.163+0000 7ffadc613200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:41:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:41:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:41:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:41:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:41:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-24T16:41:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: Failing over to other MGR 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.34931 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: mgrmap e38: y(active, starting, since 0.996887s), standbys: x 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:41:55.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: Manager daemon y is now available 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:41:55.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:54 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:41:55.894 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:55.487+0000 7ffadc613200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:41:55.894 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:55.583+0000 7ffadc613200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:41:56.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:55 smithi070 ceph-mon[26293]: [24/Apr/2024:16:41:55] ENGINE Bus STARTING 2024-04-24T16:41:56.085 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:55 smithi070 ceph-mon[26293]: mgrmap e39: y(active, since 1.9992s), standbys: x 2024-04-24T16:41:56.086 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:55 smithi070 ceph-mon[21827]: [24/Apr/2024:16:41:55] ENGINE Bus STARTING 2024-04-24T16:41:56.086 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:55 smithi070 ceph-mon[21827]: mgrmap e39: y(active, since 1.9992s), standbys: x 2024-04-24T16:41:56.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:55 smithi156 ceph-mon[22679]: [24/Apr/2024:16:41:55] ENGINE Bus STARTING 2024-04-24T16:41:56.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:55 smithi156 ceph-mon[22679]: mgrmap e39: y(active, since 1.9992s), standbys: x 2024-04-24T16:41:57.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:56.827+0000 7ffadc613200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:41:57.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:57.003+0000 7ffadc613200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:41:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-mon[22679]: [24/Apr/2024:16:41:55] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:41:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-mon[22679]: [24/Apr/2024:16:41:55] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:41:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-mon[22679]: [24/Apr/2024:16:41:55] ENGINE Bus STARTED 2024-04-24T16:41:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-mon[22679]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:41:57.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:57.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[26293]: [24/Apr/2024:16:41:55] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:41:57.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[26293]: [24/Apr/2024:16:41:55] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:41:57.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[26293]: [24/Apr/2024:16:41:55] ENGINE Bus STARTED 2024-04-24T16:41:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[26293]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:41:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[21827]: [24/Apr/2024:16:41:55] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:41:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[21827]: [24/Apr/2024:16:41:55] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:41:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[21827]: [24/Apr/2024:16:41:55] ENGINE Bus STARTED 2024-04-24T16:41:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[21827]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:41:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:57 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:41:57.458 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:57.205+0000 7ffadc613200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:41:57.821 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:57.519+0000 7ffadc613200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:41:57.821 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:57.613+0000 7ffadc613200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:41:57.821 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:57.821+0000 7ffadc613200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:41:58.144 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:58.034+0000 7ffadc613200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:41:58.144 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:58 smithi156 ceph-mon[22679]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:41:58.144 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:41:58 smithi156 ceph-mon[22679]: mgrmap e40: y(active, since 3s), standbys: x 2024-04-24T16:41:58.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:58 smithi070 ceph-mon[26293]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:41:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:41:58 smithi070 ceph-mon[26293]: mgrmap e40: y(active, since 3s), standbys: x 2024-04-24T16:41:58.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:58 smithi070 ceph-mon[21827]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:41:58.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:41:58 smithi070 ceph-mon[21827]: mgrmap e40: y(active, since 3s), standbys: x 2024-04-24T16:41:58.951 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:58.671+0000 7ffadc613200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:41:59.205 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:59.107+0000 7ffadc613200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:41:59.205 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:59.205+0000 7ffadc613200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:41:59.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:59.428+0000 7ffadc613200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:42:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[26293]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:42:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[26293]: mgrmap e41: y(active, since 5s), standbys: x 2024-04-24T16:42:00.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[21827]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[21827]: mgrmap e41: y(active, since 5s), standbys: x 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:00 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.352 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:41:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:41:59.988+0000 7ffadc613200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:42:00.353 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:00.089+0000 7ffadc613200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:42:00.353 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-mon[22679]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:42:00.353 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-mon[22679]: mgrmap e41: y(active, since 5s), standbys: x 2024-04-24T16:42:00.353 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.353 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.353 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.353 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:00.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:00.352+0000 7ffadc613200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:42:00.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:00.497+0000 7ffadc613200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:42:00.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:00.598+0000 7ffadc613200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:42:01.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:00.723+0000 7ffadc613200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:42:01.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:01.044+0000 7ffadc613200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:42:01.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:42:01.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[21827]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:42:01.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:42:01.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:42:01.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:01 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:42:01.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:01.689+0000 7ffadc613200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:42:01.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:42:01.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:42:01.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-mon[22679]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:42:02.396 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:01.914+0000 7ffadc613200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:42:02.396 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:42:01] ENGINE Bus STARTING 2024-04-24T16:42:02.396 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: CherryPy Checker: 2024-04-24T16:42:02.396 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: The Application mounted at '' has an empty config. 2024-04-24T16:42:02.396 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: 2024-04-24T16:42:02.396 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:42:02] ENGINE Serving on http://:::9283 2024-04-24T16:42:02.396 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[42886]: [24/Apr/2024:16:42:02] ENGINE Bus STARTED 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:42:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:42:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:42:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: Standby manager daemon x restarted 2024-04-24T16:42:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:42:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:42:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:42:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:42:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:02 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:42:02.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:42:02.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: Standby manager daemon x restarted 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:42:02.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-24T16:42:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: Reconfiguring daemon prometheus.a on smithi156 2024-04-24T16:42:02.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: Standby manager daemon x restarted 2024-04-24T16:42:02.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:42:02.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:42:02.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:42:02.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:42:02.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:02 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/3492761553' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:42:03.845 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:03 smithi156 ceph-mon[22679]: mgrmap e42: y(active, since 8s), standbys: x 2024-04-24T16:42:04.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:03 smithi070 ceph-mon[21827]: mgrmap e42: y(active, since 8s), standbys: x 2024-04-24T16:42:04.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:03 smithi070 ceph-mon[26293]: mgrmap e42: y(active, since 8s), standbys: x 2024-04-24T16:42:05.155 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:04 smithi156 ceph-mon[22679]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-24T16:42:05.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:04 smithi070 ceph-mon[21827]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-24T16:42:05.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:04 smithi070 ceph-mon[26293]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-24T16:42:05.409 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 systemd[1]: Stopping Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:05.409 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 bash[57400]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:42:05.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.452Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:42:05.761 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.453Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.454Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.454Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[54590]: ts=2024-04-24T16:42:05.454Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-24T16:42:05.762 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 podman[57414]: 2024-04-24 16:42:05.463270946 +0000 UTC m=+0.134926793 container died 052495724fdc11593ca3a330b66fdb3ef467a6ced5e057cd2b0fe62aeac561d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:06.094 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 podman[57414]: 2024-04-24 16:42:05.868964956 +0000 UTC m=+0.540620800 container cleanup 052495724fdc11593ca3a330b66fdb3ef467a6ced5e057cd2b0fe62aeac561d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:06.094 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 bash[57414]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a 2024-04-24T16:42:06.094 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:05 smithi156 bash[57439]: Error: no container with name or ID "ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-24T16:42:06.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 podman[57427]: 2024-04-24 16:42:06.09532733 +0000 UTC m=+0.633111773 container remove 052495724fdc11593ca3a330b66fdb3ef467a6ced5e057cd2b0fe62aeac561d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:06.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-24T16:42:06.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 systemd[1]: Stopped Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:06.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 1.393s CPU time. 2024-04-24T16:42:06.410 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 systemd[1]: Starting Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:06.411 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 podman[57510]: 2024-04-24 16:42:06.224171364 +0000 UTC m=+0.019505588 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-24T16:42:06.411 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 podman[57510]: 2024-04-24 16:42:06.337933034 +0000 UTC m=+0.133267243 container create a0a08ea29d9eff7ab66aeec7f5c28091264ba54cbae3195ae76d13ee33914821 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 podman[57510]: 2024-04-24 16:42:06.548648021 +0000 UTC m=+0.343982231 container init a0a08ea29d9eff7ab66aeec7f5c28091264ba54cbae3195ae76d13ee33914821 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 podman[57510]: 2024-04-24 16:42:06.551258354 +0000 UTC m=+0.346592565 container start a0a08ea29d9eff7ab66aeec7f5c28091264ba54cbae3195ae76d13ee33914821 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.574Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.574Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.575Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.575Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi156 (none))" 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.575Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.575Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-24T16:42:06.784 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.576Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.577Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.578Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.580Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.580Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.79µs 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.580Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 bash[57510]: a0a08ea29d9eff7ab66aeec7f5c28091264ba54cbae3195ae76d13ee33914821 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 systemd[1]: Started Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:06.785 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:06.784Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-04-24T16:42:07.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-24T16:42:07.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:42:06] ENGINE Bus STOPPING 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:42:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:42:06] ENGINE Bus STOPPED 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:42:06] ENGINE Bus STARTING 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:42:06] ENGINE Serving on http://:::9283 2024-04-24T16:42:07.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:42:06] ENGINE Bus STARTED 2024-04-24T16:42:07.159 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.022Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:06 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:07.470 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.218Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.220Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.223Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.223Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.223Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.757µs wal_replay_duration=643.125884ms total_replay_duration=643.184718ms 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.228Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.228Z caller=main.go:947 level=info msg="TSDB started" 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.228Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.251Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.029231ms db_storage=789ns remote_storage=1.25µs web_handler=491ns query_engine=756ns scrape=367.59µs scrape_sd=116.955µs notify=14.864µs notify_sd=19.37µs rules=22.175777ms 2024-04-24T16:42:07.471 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:07.251Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-24T16:42:08.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:42:08.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:07 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:42:08.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:07 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:42:08.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:07 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:42:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:07 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:42:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:07 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi156.front.sepia.ceph.com:9095"}]: dispatch 2024-04-24T16:42:09.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:08 smithi156 ceph-mon[22679]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-04-24T16:42:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:08 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:42:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:08 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:42:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:08 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:42:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:08 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[26293]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-04-24T16:42:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:42:09.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:42:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:42:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[21827]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-04-24T16:42:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:42:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:42:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:42:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:08 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:09.987 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:09 smithi156 ceph-mon[22679]: Upgrade: Updating mgr.x 2024-04-24T16:42:09.988 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:09 smithi156 ceph-mon[22679]: Deploying daemon mgr.x on smithi156 2024-04-24T16:42:09.988 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:09 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:09.988 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:09 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:10.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[21827]: Upgrade: Updating mgr.x 2024-04-24T16:42:10.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[21827]: Deploying daemon mgr.x on smithi156 2024-04-24T16:42:10.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:10.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:10.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[26293]: Upgrade: Updating mgr.x 2024-04-24T16:42:10.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[26293]: Deploying daemon mgr.x on smithi156 2024-04-24T16:42:10.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:10.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:09 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:10.557 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:10 smithi156 systemd[1]: Stopping Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:10.834 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:10 smithi156 podman[57863]: 2024-04-24 16:42:10.55712441 +0000 UTC m=+0.131612025 container died 432b981bb958d0a8730b4645c5cad0b9142cfd8c386eebc00982d477c0d4d7ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-24T16:42:11.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:10 smithi156 ceph-mon[22679]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-04-24T16:42:11.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:10 smithi156 podman[57863]: 2024-04-24 16:42:10.951926171 +0000 UTC m=+0.526413785 container cleanup 432b981bb958d0a8730b4645c5cad0b9142cfd8c386eebc00982d477c0d4d7ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:42:11.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:10 smithi156 bash[57863]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x 2024-04-24T16:42:11.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:10 smithi070 ceph-mon[21827]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-04-24T16:42:11.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:10 smithi070 ceph-mon[26293]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-04-24T16:42:11.483 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 podman[57877]: 2024-04-24 16:42:11.186621139 +0000 UTC m=+0.632368395 container remove 432b981bb958d0a8730b4645c5cad0b9142cfd8c386eebc00982d477c0d4d7ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-24T16:42:11.483 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:42:11.483 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-24T16:42:11.483 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 systemd[1]: Stopped Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:11.483 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service: Consumed 58.573s CPU time. 2024-04-24T16:42:11.483 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 systemd[1]: Starting Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:11.736 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 podman[58008]: 2024-04-24 16:42:11.603357325 +0000 UTC m=+0.018766647 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:42:11.737 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 podman[58008]: 2024-04-24 16:42:11.738035155 +0000 UTC m=+0.153444463 container create 45888cc21913fa855f0bbe574df10e47910e16ac210b710ecc196391f79e3a61 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-24T16:42:12.034 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 podman[58008]: 2024-04-24 16:42:11.957563963 +0000 UTC m=+0.372973272 container init 45888cc21913fa855f0bbe574df10e47910e16ac210b710ecc196391f79e3a61 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-24T16:42:12.034 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 podman[58008]: 2024-04-24 16:42:11.960458459 +0000 UTC m=+0.375867775 container start 45888cc21913fa855f0bbe574df10e47910e16ac210b710ecc196391f79e3a61 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-24T16:42:12.034 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:11 smithi156 ceph-mgr[58027]: -- 172.21.15.156:0/629898900 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5570949645a0 con 0x557094938800 2024-04-24T16:42:12.312 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:12 smithi156 bash[58008]: 45888cc21913fa855f0bbe574df10e47910e16ac210b710ecc196391f79e3a61 2024-04-24T16:42:12.313 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:12 smithi156 systemd[1]: Started Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:12.313 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:12.115+0000 7f056f14c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:42:12.313 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:12.312+0000 7f056f14c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:42:12.659 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:12.422+0000 7f056f14c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:42:13.102 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:12.774+0000 7f056f14c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:42:13.102 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:12.859+0000 7f056f14c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:42:13.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:13 smithi156 ceph-mon[22679]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-04-24T16:42:13.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:13 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:13.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:13 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:13.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:13 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[21827]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[26293]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:13.450 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:13 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:14.409 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:14.114+0000 7f056f14c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:42:14.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:14.302+0000 7f056f14c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:42:14.796 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:14.490+0000 7f056f14c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:42:14.796 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:14.796+0000 7f056f14c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:42:15.104 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:14.892+0000 7f056f14c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:42:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:15 smithi156 ceph-mon[22679]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:42:15.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:15.107+0000 7f056f14c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:42:15.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:15.318+0000 7f056f14c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:42:15.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:15 smithi070 ceph-mon[26293]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:42:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:15 smithi070 ceph-mon[21827]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:42:16.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:15.944+0000 7f056f14c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:42:16.613 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:16.387+0000 7f056f14c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:42:16.613 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:16.485+0000 7f056f14c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:42:16.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:42:16] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-04-24T16:42:16.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:16.715+0000 7f056f14c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:42:17.385 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:17.279+0000 7f056f14c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:42:17.385 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:17.384+0000 7f056f14c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:42:17.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-mon[22679]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:42:17.386 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:17.386 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:17.386 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:17.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[26293]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:42:17.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:17.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:17.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[21827]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-24T16:42:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:17 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:17.896 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:17.645+0000 7f056f14c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:42:17.896 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:17.790+0000 7f056f14c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:42:17.896 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:17.895+0000 7f056f14c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:42:18.159 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:18.022+0000 7f056f14c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:42:18.655 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:18.351+0000 7f056f14c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:42:18.914 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:18 smithi156 ceph-mon[22679]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:42:18.914 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:18 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:18.914 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:18 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:18 smithi070 ceph-mon[26293]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:42:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:18 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:18 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:18 smithi070 ceph-mon[21827]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-24T16:42:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:18 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:18 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:18.985+0000 7f056f14c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:19.209+0000 7f056f14c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: [24/Apr/2024:16:42:19] ENGINE Bus STARTING 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: CherryPy Checker: 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: The Application mounted at '' has an empty config. 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: [24/Apr/2024:16:42:19] ENGINE Serving on http://:::9283 2024-04-24T16:42:19.317 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: [24/Apr/2024:16:42:19] ENGINE Bus STARTED 2024-04-24T16:42:19.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-mon[22679]: Standby manager daemon x restarted 2024-04-24T16:42:19.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-mon[22679]: Standby manager daemon x started 2024-04-24T16:42:19.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:42:19.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:42:19.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:42:19.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:19 smithi156 ceph-mon[22679]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:42:20.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[26293]: Standby manager daemon x restarted 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[21827]: Standby manager daemon x restarted 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[21827]: Standby manager daemon x started 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:42:20.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:42:20.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:42:20.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:19 smithi070 ceph-mon[21827]: from='mgr.? 172.21.15.156:0/2303377181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:42:21.128 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:20 smithi156 ceph-mon[22679]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:21.128 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:20 smithi156 ceph-mon[22679]: mgrmap e43: y(active, since 26s), standbys: x 2024-04-24T16:42:21.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:20 smithi070 ceph-mon[26293]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:21.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:20 smithi070 ceph-mon[26293]: mgrmap e43: y(active, since 26s), standbys: x 2024-04-24T16:42:21.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:20 smithi070 ceph-mon[21827]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:21.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:20 smithi070 ceph-mon[21827]: mgrmap e43: y(active, since 26s), standbys: x 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-24T16:42:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:22 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:22.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:22.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:42:22.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:22.662 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:22 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all mgr 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all crash 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all mds 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all nfs 2024-04-24T16:42:23.500 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all mgr 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all crash 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all mds 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all nfs 2024-04-24T16:42:23.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:23 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all mgr 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all crash 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all mds 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all nfs 2024-04-24T16:42:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:23 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:42:23.793 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:23 smithi070 systemd[1]: Stopping Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:23.811 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:42:24.198 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 podman[74509]: 2024-04-24 16:42:24.00826216 +0000 UTC m=+0.138188161 container died a0dadc7082e4ef75ea03c8d85c0c31a2d7217a9bd049266dd15b22939921f755 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:24.501 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (13m) 24s ago 21m 23.3M - 0.23.0 ba2b418f427c 609b591dc19c 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (13m) 7s ago 20m 55.5M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 unknown 24s ago 20m 54.5M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (12s) 7s ago 25m 62.0M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (10m) 24s ago 26m 477M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (27m) 24s ago 27m 74.9M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (25m) 7s ago 25m 61.2M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (25m) 24s ago 25m 56.1M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (13m) 24s ago 21m 17.6M - 1.3.1 1dbe0e931976 a0dadc7082e4 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (13m) 7s ago 21m 15.5M - 1.3.1 1dbe0e931976 cda127a7d141 2024-04-24T16:42:24.502 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (25m) 24s ago 25m 56.2M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (24m) 24s ago 24m 56.9M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (24m) 24s ago 24m 50.6M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (23m) 24s ago 23m 51.0M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (23m) 7s ago 23m 54.6M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (23m) 7s ago 23m 52.9M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (22m) 7s ago 22m 50.5M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (22m) 7s ago 22m 55.1M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (18s) 7s ago 21m 53.1M - 2.33.4 514e6a882f6e a0a08ea29d9e 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (20m) 24s ago 20m 75.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:42:24.503 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (20m) 7s ago 20m 75.2M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:42:24.504 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 podman[74509]: 2024-04-24 16:42:24.389806934 +0000 UTC m=+0.519732921 container cleanup a0dadc7082e4ef75ea03c8d85c0c31a2d7217a9bd049266dd15b22939921f755 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:24.504 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 bash[74509]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a 2024-04-24T16:42:24.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:24 smithi070 ceph-mon[26293]: Upgrade: Updating node-exporter.a (1/2) 2024-04-24T16:42:24.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:24 smithi070 ceph-mon[26293]: Deploying daemon node-exporter.a on smithi070 2024-04-24T16:42:24.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:24 smithi070 ceph-mon[26293]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:24.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:24 smithi070 ceph-mon[21827]: Upgrade: Updating node-exporter.a (1/2) 2024-04-24T16:42:24.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:24 smithi070 ceph-mon[21827]: Deploying daemon node-exporter.a on smithi070 2024-04-24T16:42:24.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:24 smithi070 ceph-mon[21827]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:24.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:24 smithi156 ceph-mon[22679]: Upgrade: Updating node-exporter.a (1/2) 2024-04-24T16:42:24.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:24 smithi156 ceph-mon[22679]: Deploying daemon node-exporter.a on smithi070 2024-04-24T16:42:24.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:24 smithi156 ceph-mon[22679]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:24.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 podman[74543]: 2024-04-24 16:42:24.683147062 +0000 UTC m=+0.678410116 container remove a0dadc7082e4ef75ea03c8d85c0c31a2d7217a9bd049266dd15b22939921f755 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:24.792 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:42:24.918 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:42:24.918 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:42:24.918 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:42:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:42:24.920 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:42:24.920 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:42:25.200 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-24T16:42:25.200 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 systemd[1]: Stopped Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:25.200 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:24 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 5.125s CPU time. 2024-04-24T16:42:25.200 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:25 smithi070 systemd[1]: Starting Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:25.274 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:42:25.274 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:42:25.274 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:42:25.274 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-24T16:42:25.274 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-04-24T16:42:25.275 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-04-24T16:42:25.275 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:42:25.275 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-04-24T16:42:25.275 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-24T16:42:25.275 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:42:25.275 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:42:25.542 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:25 smithi070 bash[74742]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-24T16:42:25.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[21827]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:25.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[21827]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:25.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:25.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/292158962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:25.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[26293]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:25.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[26293]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:25.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:25.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:25 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/292158962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:25.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:25 smithi156 ceph-mon[22679]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:25.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:25 smithi156 ceph-mon[22679]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:25.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:25 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:25.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:25 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/292158962' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:26.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:26 smithi070 ceph-mon[26293]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:26.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:26 smithi070 ceph-mon[26293]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:26.519 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:26 smithi070 ceph-mon[21827]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:26.519 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:26 smithi070 ceph-mon[21827]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:26.519 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:26 smithi070 bash[74742]: Getting image source signatures 2024-04-24T16:42:26.520 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:26 smithi070 bash[74742]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-24T16:42:26.520 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:26 smithi070 bash[74742]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-24T16:42:26.520 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:26 smithi070 bash[74742]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-24T16:42:26.655 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:26 smithi156 ceph-mon[22679]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:26.655 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:26 smithi156 ceph-mon[22679]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:26.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:42:26] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.33.4" 2024-04-24T16:42:27.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:27 smithi070 ceph-mon[26293]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:27.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:27 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:27 smithi070 ceph-mon[26293]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:27.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:27 smithi070 ceph-mon[21827]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:27.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:27 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:27.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:27 smithi070 ceph-mon[21827]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:27.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:27 smithi156 ceph-mon[22679]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:27.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:27 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:27.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:27 smithi156 ceph-mon[22679]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:28.042 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:27 smithi070 bash[74742]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-24T16:42:28.042 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:27 smithi070 bash[74742]: Writing manifest to image destination 2024-04-24T16:42:28.303 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 podman[74742]: 2024-04-24 16:42:27.971098873 +0000 UTC m=+2.847973428 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-24T16:42:28.304 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 podman[74742]: 2024-04-24 16:42:28.086475284 +0000 UTC m=+2.963349904 container create 88827079f0f687430af3bf1d30651e16a7280562e95c51f313011fe34b3d57e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:28.720 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 podman[74742]: 2024-04-24 16:42:28.305012182 +0000 UTC m=+3.181886737 container init 88827079f0f687430af3bf1d30651e16a7280562e95c51f313011fe34b3d57e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:28.720 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 podman[74742]: 2024-04-24 16:42:28.308113626 +0000 UTC m=+3.184988190 container start 88827079f0f687430af3bf1d30651e16a7280562e95c51f313011fe34b3d57e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:28.720 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.312Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.312Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.312Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.312Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.312Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=arp 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-24T16:42:28.721 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-24T16:42:28.722 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-24T16:42:28.722 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-24T16:42:28.722 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-24T16:42:28.722 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=edac 2024-04-24T16:42:28.722 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-24T16:42:28.722 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-24T16:42:28.722 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=os 2024-04-24T16:42:28.723 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=stat 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=time 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=uname 2024-04-24T16:42:28.724 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-24T16:42:28.725 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-24T16:42:28.725 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.313Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-24T16:42:28.725 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.314Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-24T16:42:28.725 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a[74794]: ts=2024-04-24T16:42:28.314Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-24T16:42:28.725 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 bash[74742]: 88827079f0f687430af3bf1d30651e16a7280562e95c51f313011fe34b3d57e0 2024-04-24T16:42:28.725 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:42:28 smithi070 systemd[1]: Started Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:29.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:29 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:29.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:29 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:29.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:29 smithi070 ceph-mon[26293]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:29.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:29 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:29.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:29 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:29.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:29 smithi070 ceph-mon[21827]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:29.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:29 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:29 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:29 smithi156 ceph-mon[22679]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:42:30.779 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:30 smithi156 ceph-mon[22679]: Upgrade: Updating node-exporter.b (2/2) 2024-04-24T16:42:30.779 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:30 smithi156 ceph-mon[22679]: Deploying daemon node-exporter.b on smithi156 2024-04-24T16:42:30.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:30 smithi070 ceph-mon[26293]: Upgrade: Updating node-exporter.b (2/2) 2024-04-24T16:42:30.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:30 smithi070 ceph-mon[26293]: Deploying daemon node-exporter.b on smithi156 2024-04-24T16:42:30.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:30 smithi070 ceph-mon[21827]: Upgrade: Updating node-exporter.b (2/2) 2024-04-24T16:42:30.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:30 smithi070 ceph-mon[21827]: Deploying daemon node-exporter.b on smithi156 2024-04-24T16:42:31.032 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:30 smithi156 systemd[1]: Stopping Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:31.314 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 podman[59409]: 2024-04-24 16:42:31.03327779 +0000 UTC m=+0.108743733 container died cda127a7d141be473b3168aa67a355b4a02195e1554e550e904df266fc9936b8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:42:31.659 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 podman[59409]: 2024-04-24 16:42:31.431126975 +0000 UTC m=+0.506592916 container cleanup cda127a7d141be473b3168aa67a355b4a02195e1554e550e904df266fc9936b8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:42:31.660 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 bash[59409]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b 2024-04-24T16:42:31.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:31 smithi156 ceph-mon[22679]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:42:31.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:31 smithi070 ceph-mon[26293]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:42:31.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:31 smithi070 ceph-mon[21827]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:42:32.039 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 podman[59423]: 2024-04-24 16:42:31.699228872 +0000 UTC m=+0.670147477 container remove cda127a7d141be473b3168aa67a355b4a02195e1554e550e904df266fc9936b8 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:42:32.040 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:42:32.040 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-24T16:42:32.040 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 systemd[1]: Stopped Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:32.040 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:31 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 5.423s CPU time. 2024-04-24T16:42:32.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:32 smithi156 systemd[1]: Starting Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:32.409 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:32 smithi156 bash[59554]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-24T16:42:32.909 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:32 smithi156 bash[59554]: Getting image source signatures 2024-04-24T16:42:32.909 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:32 smithi156 bash[59554]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-24T16:42:32.910 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:32 smithi156 bash[59554]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-24T16:42:32.910 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:32 smithi156 bash[59554]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-24T16:42:34.159 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 bash[59554]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-24T16:42:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:33 smithi156 ceph-mon[22679]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:34.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:33 smithi070 ceph-mon[21827]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:34.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:33 smithi070 ceph-mon[26293]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:34.565 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 bash[59554]: Writing manifest to image destination 2024-04-24T16:42:34.886 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 podman[59554]: 2024-04-24 16:42:34.553000279 +0000 UTC m=+2.413795360 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-24T16:42:34.886 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 podman[59554]: 2024-04-24 16:42:34.676912129 +0000 UTC m=+2.537707194 container create 9770cba17afbc0ae7473c0ec0a024192a3763d700d7fd1081274b89fa4ed1103 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:42:35.160 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 podman[59554]: 2024-04-24 16:42:34.887228638 +0000 UTC m=+2.748023699 container init 9770cba17afbc0ae7473c0ec0a024192a3763d700d7fd1081274b89fa4ed1103 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:42:35.160 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 podman[59554]: 2024-04-24 16:42:34.890203871 +0000 UTC m=+2.750998940 container start 9770cba17afbc0ae7473c0ec0a024192a3763d700d7fd1081274b89fa4ed1103 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:42:35.160 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.894Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-24T16:42:35.160 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.894Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-24T16:42:35.160 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.894Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.894Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.894Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=arp 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-24T16:42:35.161 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=edac 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-24T16:42:35.162 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=os 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-24T16:42:35.163 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=stat 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=time 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=uname 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b[59606]: ts=2024-04-24T16:42:34.895Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-24T16:42:35.164 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 bash[59554]: 9770cba17afbc0ae7473c0ec0a024192a3763d700d7fd1081274b89fa4ed1103 2024-04-24T16:42:35.165 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:42:34 smithi156 systemd[1]: Started Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:36.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:36 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:36.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:36 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:36.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:36 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:36.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:36 smithi156 ceph-mon[22679]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:36.174 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:36.174 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:36.174 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:36.175 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[26293]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:36.175 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:36.175 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:36.175 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:36.175 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-mon[21827]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:36.786 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:42:36] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.33.4" 2024-04-24T16:42:37.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:37 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:37.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:37 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:37.353 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:37 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:38.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:38 smithi070 ceph-mon[26293]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:38.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:38 smithi070 ceph-mon[21827]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:38.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:38 smithi156 ceph-mon[22679]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:40.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[26293]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[21827]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:40 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:40.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:40 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:40 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:40 smithi156 ceph-mon[22679]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:40.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:40 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:40 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:40.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:40 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:41.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[21827]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:41 smithi070 ceph-mon[26293]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:41.909 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:41 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:41 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:41 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:41 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:41.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:41 smithi156 ceph-mon[22679]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:44.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:43 smithi156 ceph-mon[22679]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:44.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:43 smithi070 ceph-mon[26293]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:44.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:43 smithi070 ceph-mon[21827]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:44 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:45.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.278 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:45.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:44 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:46.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:42:46] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-04-24T16:42:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:47 smithi156 ceph-mon[22679]: Upgrade: Updating prometheus.a 2024-04-24T16:42:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:47 smithi156 ceph-mon[22679]: Deploying daemon prometheus.a on smithi156 2024-04-24T16:42:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:47 smithi156 ceph-mon[22679]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:47 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[26293]: Upgrade: Updating prometheus.a 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[26293]: Deploying daemon prometheus.a on smithi156 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[26293]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[21827]: Upgrade: Updating prometheus.a 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[21827]: Deploying daemon prometheus.a on smithi156 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[21827]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:47 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:49.159 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:48 smithi156 ceph-mon[22679]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:49.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:48 smithi070 ceph-mon[26293]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:48 smithi070 ceph-mon[21827]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:51 smithi156 ceph-mon[22679]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 582 B/s rd, 0 op/s 2024-04-24T16:42:51.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:51 smithi070 ceph-mon[26293]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 582 B/s rd, 0 op/s 2024-04-24T16:42:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:51 smithi070 ceph-mon[21827]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 582 B/s rd, 0 op/s 2024-04-24T16:42:53.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:53 smithi156 ceph-mon[22679]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:53.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:53 smithi070 ceph-mon[26293]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:53.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:53 smithi070 ceph-mon[21827]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:42:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:55 smithi156 ceph-mon[22679]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:42:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:55 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:55 smithi070 ceph-mon[26293]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:42:55.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:55 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:55 smithi070 ceph-mon[21827]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:42:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:55 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:42:55.636 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:42:56.290 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (13m) 16s ago 21m 23.3M - 0.23.0 ba2b418f427c 609b591dc19c 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (13m) 17s ago 21m 55.8M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 unknown 16s ago 20m 54.9M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (44s) 17s ago 25m 450M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (10m) 16s ago 27m 501M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (27m) 16s ago 27m 77.0M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (26m) 17s ago 26m 61.2M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (26m) 16s ago 26m 58.6M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (28s) 16s ago 22m 3586k - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (21s) 17s ago 22m 3602k - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (25m) 16s ago 25m 57.0M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (25m) 16s ago 25m 58.1M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (24m) 16s ago 24m 50.9M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:42:56.291 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (24m) 16s ago 24m 52.2M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:42:56.292 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (24m) 17s ago 24m 54.6M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:42:56.292 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (23m) 17s ago 23m 52.9M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:42:56.292 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (23m) 17s ago 23m 50.5M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:42:56.292 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (23m) 17s ago 23m 55.1M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:42:56.292 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (50s) 17s ago 22m 57.5M - 2.33.4 514e6a882f6e a0a08ea29d9e 2024-04-24T16:42:56.292 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (20m) 16s ago 20m 75.3M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:42:56.292 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (20m) 17s ago 20m 75.3M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:42:56.690 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:42:56.690 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:42:56.691 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:42:56.692 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:42:56.692 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:42:56.692 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:42:56.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:42:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:42:56] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-04-24T16:42:57.022 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-24T16:42:57.023 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:42:57.023 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:42:57.283 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:57 smithi156 ceph-mon[22679]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:57.284 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:57 smithi156 ceph-mon[22679]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:57.284 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:57 smithi156 ceph-mon[22679]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:57.284 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:57 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:57.284 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:57 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/2712372588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:57.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[26293]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:57.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[26293]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:57.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[26293]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/2712372588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[21827]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[21827]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[21827]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:42:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:42:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:57 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/2712372588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:42:58.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-mon[22679]: from='client.25510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:58.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-mon[22679]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:58.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:58 smithi070 ceph-mon[21827]: from='client.25510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:58.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:58 smithi070 ceph-mon[21827]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:58.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:58 smithi070 ceph-mon[26293]: from='client.25510 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:58.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:58 smithi070 ceph-mon[26293]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:42:58.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 systemd[1]: Stopping Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:58.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:42:58.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-24T16:42:58.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-24T16:42:58.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-24T16:42:58.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-24T16:42:58.660 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-24T16:42:58.661 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-24T16:42:58.661 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-24T16:42:58.661 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-24T16:42:58.661 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-24T16:42:58.661 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-24T16:42:58.661 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[57525]: ts=2024-04-24T16:42:58.584Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-24T16:42:58.661 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 podman[61105]: 2024-04-24 16:42:58.594452151 +0000 UTC m=+0.106114150 container died a0a08ea29d9eff7ab66aeec7f5c28091264ba54cbae3195ae76d13ee33914821 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:59.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 podman[61105]: 2024-04-24 16:42:58.967311916 +0000 UTC m=+0.478973908 container cleanup a0a08ea29d9eff7ab66aeec7f5c28091264ba54cbae3195ae76d13ee33914821 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:59.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:58 smithi156 bash[61105]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a 2024-04-24T16:42:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:42:59 smithi156 ceph-mon[22679]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:59.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:42:59 smithi070 ceph-mon[21827]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:59.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:42:59 smithi070 ceph-mon[26293]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:42:59.601 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:59 smithi156 podman[61119]: 2024-04-24 16:42:59.251925149 +0000 UTC m=+0.657960967 container remove a0a08ea29d9eff7ab66aeec7f5c28091264ba54cbae3195ae76d13ee33914821 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:42:59.601 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:59 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-24T16:42:59.602 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:59 smithi156 systemd[1]: Stopped Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:42:59.602 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:59 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 1.288s CPU time. 2024-04-24T16:42:59.910 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:59 smithi156 systemd[1]: Starting Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:42:59.910 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:42:59 smithi156 podman[61252]: 2024-04-24 16:42:59.722092104 +0000 UTC m=+0.019480638 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-04-24T16:43:00.910 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:00 smithi156 podman[61252]: 2024-04-24 16:43:00.553917398 +0000 UTC m=+0.851305917 container create f240eb7d50fac207ebc3c6697f3b67c79ad9eddca1d1ccc66ef4ffa9e6e11ba3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-mon[22679]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:01.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:01 smithi070 ceph-mon[26293]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:01.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:01 smithi070 ceph-mon[21827]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:01.910 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 podman[61252]: 2024-04-24 16:43:01.540246409 +0000 UTC m=+1.837634928 container init f240eb7d50fac207ebc3c6697f3b67c79ad9eddca1d1ccc66ef4ffa9e6e11ba3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:01.910 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 podman[61252]: 2024-04-24 16:43:01.543123211 +0000 UTC m=+1.840511731 container start f240eb7d50fac207ebc3c6697f3b67c79ad9eddca1d1ccc66ef4ffa9e6e11ba3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:01.910 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.569Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.569Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.569Z caller=main.go:570 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi156 (none))" 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.569Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.569Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.570Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.571Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.572Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.572Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.574Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.574Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.188µs 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.574Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-24T16:43:01.911 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.580Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.595Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.605Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.607Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.609Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.611Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.611Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.611Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.913µs wal_replay_duration=36.597065ms wbl_replay_duration=116ns total_replay_duration=36.64717ms 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.615Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.615Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.615Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 bash[61252]: f240eb7d50fac207ebc3c6697f3b67c79ad9eddca1d1ccc66ef4ffa9e6e11ba3 2024-04-24T16:43:01.912 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 systemd[1]: Started Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:43:01.913 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.637Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.075461ms db_storage=996ns remote_storage=1.261µs web_handler=441ns query_engine=670ns scrape=485.29µs scrape_sd=107.229µs notify=17.777µs notify_sd=17.653µs rules=21.115922ms tracing=5.822µs 2024-04-24T16:43:01.913 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.637Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-24T16:43:01.913 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:43:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:43:01.637Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-24T16:43:02.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:02 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:02.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:02 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:02.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:02 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:02.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:02 smithi156 ceph-mon[22679]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:03.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:03.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:03.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:03.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[26293]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:03.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:03.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:03.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:03.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:02 smithi070 ceph-mon[21827]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:05 smithi156 ceph-mon[22679]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:05.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:05 smithi070 ceph-mon[26293]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:05 smithi070 ceph-mon[21827]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[26293]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[21827]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:06 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:07.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:06 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:06 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:06 smithi156 ceph-mon[22679]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:06 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:08.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:08 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:08.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:08 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:08 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:08 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:08 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:08 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:09 smithi156 ceph-mon[22679]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:09 smithi070 ceph-mon[26293]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:09 smithi070 ceph-mon[21827]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:10.279 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:10 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:10.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:10 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:10.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:10 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:11 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:11.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:11.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:11 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:12 smithi070 ceph-mon[21827]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-04-24T16:43:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:12 smithi070 ceph-mon[21827]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-04-24T16:43:12.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:12 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:43:12.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:12 smithi070 ceph-mon[26293]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-04-24T16:43:12.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:12 smithi070 ceph-mon[26293]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-04-24T16:43:12.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:12 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:43:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:12 smithi156 ceph-mon[22679]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2024-04-24T16:43:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:12 smithi156 ceph-mon[22679]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-04-24T16:43:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:12 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-24T16:43:13.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:13 smithi070 ceph-mon[26293]: Upgrade: Updating alertmanager.a 2024-04-24T16:43:13.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:13 smithi070 ceph-mon[26293]: Deploying daemon alertmanager.a on smithi070 2024-04-24T16:43:13.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:13 smithi070 ceph-mon[21827]: Upgrade: Updating alertmanager.a 2024-04-24T16:43:13.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:13 smithi070 ceph-mon[21827]: Deploying daemon alertmanager.a on smithi070 2024-04-24T16:43:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:13 smithi156 ceph-mon[22679]: Upgrade: Updating alertmanager.a 2024-04-24T16:43:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:13 smithi156 ceph-mon[22679]: Deploying daemon alertmanager.a on smithi070 2024-04-24T16:43:14.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:14 smithi070 ceph-mon[26293]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:14.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:14 smithi070 ceph-mon[21827]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:14 smithi156 ceph-mon[22679]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:16.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:16 smithi070 ceph-mon[26293]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-04-24T16:43:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:16 smithi070 ceph-mon[21827]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-04-24T16:43:16.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:16 smithi156 ceph-mon[22679]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-04-24T16:43:16.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:43:16] "GET /metrics HTTP/1.1" 200 37989 "" "Prometheus/2.43.0" 2024-04-24T16:43:17.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:17 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:17 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:17.376 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:17 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:18.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:18 smithi070 ceph-mon[26293]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:18.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:18 smithi070 ceph-mon[21827]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:18 smithi156 ceph-mon[22679]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:20.166 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:20 smithi070 ceph-mon[26293]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:20.166 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:20 smithi070 ceph-mon[21827]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:20.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:20 smithi156 ceph-mon[22679]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:20.792 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:20 smithi070 systemd[1]: Stopping Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:43:21.121 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:20 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[55877]: level=info ts=2024-04-24T16:43:20.813Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:43:21.122 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:20 smithi070 podman[76485]: 2024-04-24 16:43:20.854617314 +0000 UTC m=+0.174410901 container died 609b591dc19cf80f9834141824ed44d8127ae4b6a9db6ed649674afeef3498bb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:21.542 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:21 smithi070 podman[76485]: 2024-04-24 16:43:21.30537278 +0000 UTC m=+0.625166366 container cleanup 609b591dc19cf80f9834141824ed44d8127ae4b6a9db6ed649674afeef3498bb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:21.542 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:21 smithi070 bash[76485]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a 2024-04-24T16:43:22.261 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:22 smithi070 ceph-mon[26293]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:43:22.261 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:22 smithi070 ceph-mon[21827]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:43:22.261 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:21 smithi070 podman[76500]: 2024-04-24 16:43:21.965266866 +0000 UTC m=+1.105979772 container remove 609b591dc19cf80f9834141824ed44d8127ae4b6a9db6ed649674afeef3498bb (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:22.261 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 podman[76500]: 2024-04-24 16:43:22.065972587 +0000 UTC m=+1.206685448 volume remove e674ffab6f58e438eaf93c5d30af1f55e52fc391452dd13b3f59b1b8a41fda67 2024-04-24T16:43:22.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:22 smithi156 ceph-mon[22679]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:43:22.542 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-24T16:43:22.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 systemd[1]: Stopped Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:43:22.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service: Consumed 1.482s CPU time. 2024-04-24T16:43:22.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 systemd[1]: Starting Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:43:22.961 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 podman[76629]: 2024-04-24 16:43:22.700304051 +0000 UTC m=+0.114635731 volume create d8fe49e43498d6b2af14cde3c8e469e647f89528c8bce001794b8c63dc913f00 2024-04-24T16:43:22.962 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 podman[76629]: 2024-04-24 16:43:22.603654948 +0000 UTC m=+0.017986586 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-24T16:43:22.962 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:22 smithi070 podman[76629]: 2024-04-24 16:43:22.783916579 +0000 UTC m=+0.198248218 container create a84220aabb99879cecf9bd73d86bf9b4f3fe918426a17a37d44c51128e2d8f97 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:23.292 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 podman[76629]: 2024-04-24 16:43:23.053725511 +0000 UTC m=+0.468057148 container init a84220aabb99879cecf9bd73d86bf9b4f3fe918426a17a37d44c51128e2d8f97 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 podman[76629]: 2024-04-24 16:43:23.056423546 +0000 UTC m=+0.470755188 container start a84220aabb99879cecf9bd73d86bf9b4f3fe918426a17a37d44c51128e2d8f97 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.068Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.068Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.070Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.70 port=9094 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.071Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.094Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.094Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.097Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:23.097Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 bash[76629]: a84220aabb99879cecf9bd73d86bf9b4f3fe918426a17a37d44c51128e2d8f97 2024-04-24T16:43:23.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:23 smithi070 systemd[1]: Started Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:43:24.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:24 smithi156 ceph-mon[22679]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:24.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:24 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:24.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:24 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:24.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:24 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:24.483 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[26293]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:24.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:24.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:24.484 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:24.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[21827]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:24.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:24.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:24.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:24 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:25.213 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:25.071Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000194751s 2024-04-24T16:43:25.537 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:25 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:25.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:25 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:25.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:25 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:26.492 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:26 smithi156 ceph-mon[22679]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:26.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:26 smithi070 ceph-mon[26293]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:26.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:26 smithi070 ceph-mon[21827]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:26.520 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:43:26] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.43.0" 2024-04-24T16:43:27.386 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:43:27.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:27 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:27.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:27 smithi070 ceph-mon[26293]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:27.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:27 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:27.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:27 smithi070 ceph-mon[21827]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:27.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:27 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:27 smithi156 ceph-mon[22679]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:28.075 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:43:28.075 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (5s) 0s ago 22m 16.7M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (14m) 0s ago 21m 57.0M - 8.3.5 dad864ee21e9 3fc0342b4717 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 unknown 0s ago 21m 58.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (76s) 0s ago 26m 450M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (11m) 0s ago 28m 503M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (28m) 0s ago 28m 78.1M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (26m) 0s ago 26m 61.2M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (26m) 0s ago 26m 56.3M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (60s) 0s ago 22m 8568k - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (53s) 0s ago 22m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (26m) 0s ago 26m 57.0M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (25m) 0s ago 25m 58.1M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:43:28.076 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (25m) 0s ago 25m 50.9M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (25m) 0s ago 25m 52.2M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (24m) 0s ago 24m 54.6M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (24m) 0s ago 24m 52.9M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (23m) 0s ago 23m 50.5M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (23m) 0s ago 23m 55.1M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (28s) 0s ago 22m 53.5M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (21m) 0s ago 21m 75.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:43:28.077 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (21m) 0s ago 21m 75.5M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:43:28.491 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:43:28.492 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:43:28.493 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:43:28.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[21827]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.792 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[21827]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[21827]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[26293]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[26293]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:28 smithi070 ceph-mon[26293]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.849 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:43:28.850 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:43:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:28 smithi156 ceph-mon[22679]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:28 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:28 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:28 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:28 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:28 smithi156 ceph-mon[22679]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:28 smithi156 ceph-mon[22679]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:29.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[21827]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:29.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/213186573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:29.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:29.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:29.618 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[21827]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:29.618 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[26293]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:29.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/213186573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:29.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:29.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:29.619 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:29 smithi070 ceph-mon[26293]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:29 smithi156 ceph-mon[22679]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:29 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/213186573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:29 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:29 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:29 smithi156 ceph-mon[22679]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:43:31.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:31 smithi070 ceph-mon[26293]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:31.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:31 smithi070 ceph-mon[21827]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:31.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:31 smithi156 ceph-mon[22679]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:32.503 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:32] ENGINE Bus STOPPING 2024-04-24T16:43:32.503 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:43:32.503 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:32] ENGINE Bus STOPPED 2024-04-24T16:43:32.503 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:32] ENGINE Bus STARTING 2024-04-24T16:43:32.503 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:32] ENGINE Serving on http://:::9283 2024-04-24T16:43:32.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:32] ENGINE Bus STARTED 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:43:33.250 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070:9093"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070:9093"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.251 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:33 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.337 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:43:33.075Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003880012s 2024-04-24T16:43:33.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:33.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:33.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:33.337 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070:9093"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070:9093"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.338 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070:9093"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi070:9093"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:33.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:33 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:34 smithi070 ceph-mon[26293]: Upgrade: Updating grafana.a 2024-04-24T16:43:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:34 smithi070 ceph-mon[26293]: Deploying daemon grafana.a on smithi156 2024-04-24T16:43:34.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:34 smithi070 ceph-mon[21827]: Upgrade: Updating grafana.a 2024-04-24T16:43:34.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:34 smithi070 ceph-mon[21827]: Deploying daemon grafana.a on smithi156 2024-04-24T16:43:34.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:34 smithi156 ceph-mon[22679]: Upgrade: Updating grafana.a 2024-04-24T16:43:34.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:34 smithi156 ceph-mon[22679]: Deploying daemon grafana.a on smithi156 2024-04-24T16:43:35.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:35 smithi070 ceph-mon[26293]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:35.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:35 smithi070 ceph-mon[21827]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:35.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:35 smithi156 ceph-mon[22679]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:36.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:43:36] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.43.0" 2024-04-24T16:43:37.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:37 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:37.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:37 smithi070 ceph-mon[26293]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:37.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:37 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:37.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:37 smithi070 ceph-mon[21827]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:37.659 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:37 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:37.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:37 smithi156 ceph-mon[22679]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:40.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:39 smithi070 ceph-mon[26293]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:40.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:39 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:40.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:39 smithi070 ceph-mon[21827]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:40.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:39 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:40.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:39 smithi156 ceph-mon[22679]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:40.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:39 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:42.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:41 smithi156 ceph-mon[22679]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:42.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:41 smithi070 ceph-mon[26293]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:42.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:41 smithi070 ceph-mon[21827]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:43:44.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:43 smithi070 ceph-mon[26293]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:43:44.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:43 smithi070 ceph-mon[21827]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:43:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:43 smithi156 ceph-mon[22679]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:43:44.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:43 smithi156 systemd[1]: Stopping Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:43:44.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[46581]: t=2024-04-24T16:43:44+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-24T16:43:44.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 podman[63326]: 2024-04-24 16:43:44.167975708 +0000 UTC m=+0.106837285 container died 3fc0342b47172515d6275e2fd0889184f5b8fe1b235e12b7217e6b31f0e7d062 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, architecture=x86_64, name=ubi8, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, distribution-scope=public, version=8.5) 2024-04-24T16:43:44.783 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 podman[63326]: 2024-04-24 16:43:44.573998526 +0000 UTC m=+0.512860149 container cleanup 3fc0342b47172515d6275e2fd0889184f5b8fe1b235e12b7217e6b31f0e7d062 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.openshift.tags=base rhel8, vcs-type=git, vendor=Red Hat, Inc., architecture=x86_64, name=ubi8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.buildah.version=1.24.2, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-24T16:43:44.783 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 bash[63326]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a 2024-04-24T16:43:45.074 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 podman[63340]: 2024-04-24 16:43:44.78396743 +0000 UTC m=+0.613991684 container remove 3fc0342b47172515d6275e2fd0889184f5b8fe1b235e12b7217e6b31f0e7d062 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, name=ubi8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-24T16:43:45.074 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-24T16:43:45.074 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 systemd[1]: Stopped Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:43:45.074 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:44 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@grafana.a.service: Consumed 5.552s CPU time. 2024-04-24T16:43:45.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 systemd[1]: Starting Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:43:45.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 podman[63470]: 2024-04-24 16:43:45.192456918 +0000 UTC m=+0.018706544 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-04-24T16:43:45.410 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 podman[63470]: 2024-04-24 16:43:45.318902766 +0000 UTC m=+0.145152384 container create 135413e06c8551842ce753383a4490a5cbba6106687257f3abeb1c1a137da745 (image=quay.io/ceph/grafana:9.4.12, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-24T16:43:45.700 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 podman[63470]: 2024-04-24 16:43:45.546062155 +0000 UTC m=+0.372311772 container init 135413e06c8551842ce753383a4490a5cbba6106687257f3abeb1c1a137da745 (image=quay.io/ceph/grafana:9.4.12, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-24T16:43:45.700 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 podman[63470]: 2024-04-24 16:43:45.548731356 +0000 UTC m=+0.374980983 container start 135413e06c8551842ce753383a4490a5cbba6106687257f3abeb1c1a137da745 (image=quay.io/ceph/grafana:9.4.12, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-24T16:43:45.700 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 bash[63470]: 135413e06c8551842ce753383a4490a5cbba6106687257f3abeb1c1a137da745 2024-04-24T16:43:45.700 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 systemd[1]: Started Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:43:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-mon[22679]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701353592Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701555681Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701613984Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.70164591Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701672898Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701682959Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701687763Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701692052Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-24T16:43:46.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701696499Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.70170176Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701707208Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701711851Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701716338Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701720939Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701725512Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701729959Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701734467Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.70173892Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=settings t=2024-04-24T16:43:45.701743617Z level=info msg="App mode production" 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=sqlstore t=2024-04-24T16:43:45.701803547Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-24T16:43:46.162 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=sqlstore t=2024-04-24T16:43:45.701838549Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.716582456Z level=info msg="Starting DB migrations" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.719808687Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.731093188Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.732244932Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.732553532Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.733769722Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.734798158Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.735806079Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.736102476Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.736573605Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.737065489Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-24T16:43:46.163 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.737412734Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.73848426Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.738799943Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.739808873Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.7408562Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.741897036Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.742176504Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.74319284Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.744250522Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.744513746Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.745554258Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.746588271Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-24T16:43:46.164 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.746999577Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.747415961Z level=info msg="Executing migration" id="create alert_image table" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.747806106Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.748219999Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.748453813Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.748821252Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.749217873Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.749289863Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.749491011Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.749925864Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-24T16:43:46.165 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.750332953Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-24T16:43:46.166 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.750569879Z level=info msg="Executing migration" id="create secrets table" 2024-04-24T16:43:46.166 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.750945983Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-24T16:43:46.166 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.757021612Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-24T16:43:46.166 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.758141454Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.758411527Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.764851906Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.7711572Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.772350539Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.772833773Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.77328463Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.773537637Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.773986348Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.775499129Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.776218519Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.776522535Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-24T16:43:46.167 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.776847144Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.777243086Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.777698406Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.778801503Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.77914857Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.779590414Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.780012975Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.780451198Z level=info msg="Executing migration" id="add correlation config column" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.781559754Z level=info msg="Executing migration" id="create entity_events table" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.781953853Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.782361015Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.782449105Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.782670962Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-24T16:43:46.168 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.782746127Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.783037379Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.783471302Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.783874946Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.784350886Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.78477803Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.785221709Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.785630589Z level=info msg="Executing migration" id="Drop public config table" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.786072991Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.786493632Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-24T16:43:46.169 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.786950492Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.787418468Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.787856321Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.791747368Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.792940654Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.794074986Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.794384325Z level=info msg="Executing migration" id="add share column" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.795555858Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.79605457Z level=info msg="Executing migration" id="create file table" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.79645236Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.79690256Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.797358732Z level=info msg="Executing migration" id="create file_meta table" 2024-04-24T16:43:46.170 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.79774275Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.798277443Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.798562128Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.799628502Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.800262685Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.800864871Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.802054349Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.802355936Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.802800205Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.80320245Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.803698288Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.804087849Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.805277406Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.806494695Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-24T16:43:46.171 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.806968898Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.820462751Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.820957374Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.821398037Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.825868082Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.826402896Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.826713476Z level=info msg="Executing migration" id="create folder table" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.827106724Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.827538975Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.827984659Z level=info msg="Executing migration" id="Update folder title length" 2024-04-24T16:43:46.172 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.828260085Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-24T16:43:46.173 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=migrator t=2024-04-24T16:43:45.828701577Z level=info msg="migrations completed" performed=105 skipped=376 duration=108.929643ms 2024-04-24T16:43:46.173 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=sqlstore t=2024-04-24T16:43:45.829046851Z level=info msg="Created default organization" 2024-04-24T16:43:46.173 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=plugin.finder t=2024-04-24T16:43:45.862781842Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-24T16:43:46.173 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=secrets t=2024-04-24T16:43:45.862907342Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-24T16:43:46.173 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=query_data t=2024-04-24T16:43:45.865112988Z level=info msg="Query Service initialization" 2024-04-24T16:43:46.173 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:45 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=live.push_http t=2024-04-24T16:43:45.867942761Z level=info msg="Live Push Gateway initialization" 2024-04-24T16:43:46.234 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[26293]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:46.234 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:46.234 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:46.234 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:46.234 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[21827]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:46.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:46.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:46.235 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:45 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:43:46] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.43.0" 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=infra.usagestats.collector t=2024-04-24T16:43:46.522211718Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=provisioning.datasources t=2024-04-24T16:43:46.523996863Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=provisioning.datasources t=2024-04-24T16:43:46.524005343Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=provisioning.datasources t=2024-04-24T16:43:46.524100255Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=provisioning.alerting t=2024-04-24T16:43:46.546074894Z level=info msg="starting to provision alerting" 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=provisioning.alerting t=2024-04-24T16:43:46.546086015Z level=info msg="finished to provision alerting" 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=grafanaStorageLogger t=2024-04-24T16:43:46.546253442Z level=info msg="storage starting" 2024-04-24T16:43:46.834 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=ngalert.state.manager t=2024-04-24T16:43:46.546313803Z level=info msg="Warming state cache for startup" 2024-04-24T16:43:46.835 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=ngalert.state.manager t=2024-04-24T16:43:46.552476389Z level=info msg="State cache has been initialized" states=0 duration=6.160557ms 2024-04-24T16:43:46.835 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=ngalert.multiorg.alertmanager t=2024-04-24T16:43:46.552776287Z level=info msg="starting MultiOrg Alertmanager" 2024-04-24T16:43:46.835 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=ticker t=2024-04-24T16:43:46.553497791Z level=info msg=starting first_tick=2024-04-24T16:43:50Z 2024-04-24T16:43:46.835 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=http.server t=2024-04-24T16:43:46.554785201Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-24T16:43:46.835 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=sqlstore.transactions t=2024-04-24T16:43:46.565482132Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-24T16:43:46.835 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:43:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=sqlstore.transactions t=2024-04-24T16:43:46.580053713Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-24T16:43:48.256 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:47 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:48.256 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:47 smithi156 ceph-mon[22679]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:48.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:47 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:48.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:47 smithi070 ceph-mon[26293]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:48.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:47 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:48.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:47 smithi070 ceph-mon[21827]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:50.205 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:49 smithi156 ceph-mon[22679]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:50.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:49 smithi070 ceph-mon[21827]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:50.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:49 smithi070 ceph-mon[26293]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:51.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:51 smithi156 ceph-mon[22679]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:51.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:51 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:51.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:51 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:52.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:51 smithi070 ceph-mon[21827]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:52.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:51 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:52.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:51 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:52.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:51 smithi070 ceph-mon[26293]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:52.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:51 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:52.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:51 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:52.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:52 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:52.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:52 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:53.042 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:52 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:53.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:52 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:52 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:53.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:52 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:54.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:53 smithi070 ceph-mon[26293]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:53 smithi070 ceph-mon[21827]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:54.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:53 smithi156 ceph-mon[22679]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:55.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:55.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:54 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:43:56.243 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:55 smithi070 ceph-mon[26293]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:56.243 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:55 smithi070 ceph-mon[21827]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:55 smithi156 ceph-mon[22679]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:43:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:43:56] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.43.0" 2024-04-24T16:43:58.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:57] ENGINE Bus STOPPING 2024-04-24T16:43:58.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:43:58.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:57] ENGINE Bus STOPPED 2024-04-24T16:43:58.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:57] ENGINE Bus STARTING 2024-04-24T16:43:58.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:57] ENGINE Serving on http://:::9283 2024-04-24T16:43:58.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:43:57] ENGINE Bus STARTED 2024-04-24T16:43:58.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156.front.sepia.ceph.com:3000"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:43:58.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156.front.sepia.ceph.com:3000"}]: dispatch 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.047 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:43:58.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:43:58.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:43:58.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:43:58.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:43:58.054 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:43:58.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:43:58.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:43:58.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:58.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:58.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:58.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:57 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.392 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156.front.sepia.ceph.com:3000"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.393 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:43:58.394 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:43:58.395 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.396 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:43:58.397 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:43:58.398 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:58.398 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:58.398 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:58.398 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:57 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156.front.sepia.ceph.com:3000"}]: dispatch 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: Upgrade: Finalizing container_image settings 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: Upgrade: Complete! 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:59.208 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156.front.sepia.ceph.com:3000"}]: dispatch 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: Upgrade: Finalizing container_image settings 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: Upgrade: Complete! 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-04-24T16:43:59.209 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:59.210 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:59.210 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:59.210 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:59.210 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:43:59.210 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:58 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 898 B/s rd, 0 op/s 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi156.front.sepia.ceph.com:3000"}]: dispatch 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: Upgrade: Finalizing container_image settings 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: Upgrade: Complete! 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:43:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:43:59.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:43:59.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:58 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:43:59.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-24T16:44:00.206 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:59 smithi070 ceph-mon[26293]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:00.207 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:43:59 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:00.207 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:59 smithi070 ceph-mon[21827]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:00.207 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:43:59 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:59 smithi156 ceph-mon[22679]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:43:59 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:01.122 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:00 smithi070 ceph-mon[26293]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:01.122 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:00 smithi070 ceph-mon[21827]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:00 smithi156 ceph-mon[22679]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:01.917 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:44:02.203 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:01 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/1746200462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:02.204 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:01 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/1746200462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:01 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/1746200462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:02.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-24T16:44:03.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:02 smithi070 ceph-mon[21827]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:44:03.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:02 smithi070 ceph-mon[26293]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:44:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:02 smithi156 ceph-mon[22679]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:44:04.540 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)" 2024-04-24T16:44:05.144 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:05 smithi070 ceph-mon[21827]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:44:05.145 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:05 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/859171988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:05.145 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:05 smithi070 ceph-mon[26293]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:44:05.145 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:05 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/859171988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:05.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-24T16:44:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:05 smithi156 ceph-mon[22679]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:44:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:05 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/859171988' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:06.681 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:44:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:44:06] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.43.0" 2024-04-24T16:44:07.188 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:44:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:07 smithi156 ceph-mon[22679]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:07.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:07 smithi070 ceph-mon[26293]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:07.477 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:07 smithi070 ceph-mon[21827]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:07.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-24T16:44:08.347 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:08 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:08.347 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:08 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/946556245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:08.348 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:08 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:08.348 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:08 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/946556245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:08 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:08 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/946556245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:09.279 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:09 smithi070 ceph-mon[26293]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:09.279 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:09 smithi070 ceph-mon[21827]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:09 smithi156 ceph-mon[22679]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:10.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:10 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:10.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:10 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:10.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:10 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:11.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:11 smithi070 ceph-mon[21827]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:11.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:11 smithi070 ceph-mon[21827]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:11 smithi070 ceph-mon[26293]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:11 smithi070 ceph-mon[26293]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:11.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:11 smithi156 ceph-mon[22679]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:11.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:11 smithi156 ceph-mon[22679]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:11.795 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:44:12.463 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:44:13.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:13 smithi156 ceph-mon[22679]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:13.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:13 smithi070 ceph-mon[26293]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:13.424 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:13 smithi070 ceph-mon[21827]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:14.411 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:44:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": null, 2024-04-24T16:44:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": false, 2024-04-24T16:44:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "", 2024-04-24T16:44:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:44:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": null, 2024-04-24T16:44:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-04-24T16:44:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:44:14.412 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:44:15.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:44:15.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:15 smithi070 ceph-mon[26293]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:15.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:15 smithi070 ceph-mon[21827]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:15 smithi156 ceph-mon[22679]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:16.257 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:16 smithi070 ceph-mon[21827]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:16.258 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:16 smithi070 ceph-mon[26293]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:16.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:16 smithi156 ceph-mon[22679]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:44:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:44:16] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.43.0" 2024-04-24T16:44:17.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:17 smithi070 ceph-mon[26293]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:17.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:17 smithi070 ceph-mon[21827]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:17.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:17 smithi156 ceph-mon[22679]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:17.363 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:17.363 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:17.363 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:44:17.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-04-24T16:44:18.240 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:18 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:18.241 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:18 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/810689391' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:44:18.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:18 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:18.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:18 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/810689391' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:44:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:18 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:18 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/810689391' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:44:19.205 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:19 smithi070 ceph-mon[26293]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:19.205 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:19 smithi070 ceph-mon[21827]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:19 smithi156 ceph-mon[22679]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:21.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:21 smithi070 ceph-mon[26293]: from='client.25600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:21.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:21 smithi070 ceph-mon[26293]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:21.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:21 smithi070 ceph-mon[21827]: from='client.25600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:21.388 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:21 smithi070 ceph-mon[21827]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:21.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:21 smithi156 ceph-mon[22679]: from='client.25600 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:21.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:21 smithi156 ceph-mon[22679]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:22.393 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:22 smithi070 ceph-mon[26293]: from='client.25606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mon", "hosts": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:22.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:22 smithi070 ceph-mon[21827]: from='client.25606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mon", "hosts": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:22.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:22 smithi156 ceph-mon[22679]: from='client.25606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mon", "hosts": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:22.433 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:44:23.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-24T16:44:23.395 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[21827]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:23.395 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:23.395 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:23.395 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:23.395 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:23.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:23.396 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[26293]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:23.396 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:23.396 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:23.396 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:23.396 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:23.396 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:23 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:23 smithi156 ceph-mon[22679]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:23 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:23 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:23 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:23 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:23 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:24.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[21827]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:44:24.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[21827]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s 2024-04-24T16:44:24.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[21827]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:24.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[21827]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:44:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[26293]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:44:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[26293]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s 2024-04-24T16:44:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[26293]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:24 smithi070 ceph-mon[26293]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:44:24.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:24 smithi156 ceph-mon[22679]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:44:24.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:24 smithi156 ceph-mon[22679]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s 2024-04-24T16:44:24.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:24 smithi156 ceph-mon[22679]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:24.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:24 smithi156 ceph-mon[22679]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:44:25.260 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:44:25.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:25 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:25.340 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:25 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:25 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:25.940 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:44:25.940 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (63s) 58s ago 23m 16.7M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:44:25.940 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (40s) 35s ago 22m 51.6M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:44:25.941 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 unknown 58s ago 22m 58.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:44:25.941 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (2m) 35s ago 27m 451M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:44:25.941 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (12m) 58s ago 29m 503M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:44:25.941 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (29m) 58s ago 29m 78.1M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:44:25.941 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (27m) 35s ago 27m 61.3M 2048M 17.2.0 e1d6a67b021e 9f6c28bccd1d 2024-04-24T16:44:25.941 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (27m) 58s ago 27m 56.3M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:44:25.941 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (117s) 58s ago 23m 8568k - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:44:25.942 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (111s) 35s ago 23m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:44:25.942 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (27m) 58s ago 27m 57.0M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:44:25.942 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (26m) 58s ago 26m 58.1M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:44:25.942 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (26m) 58s ago 26m 50.9M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:44:25.942 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (25m) 58s ago 25m 52.2M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:44:25.942 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (25m) 35s ago 25m 54.6M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:44:25.943 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (25m) 35s ago 25m 52.9M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:44:25.943 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (24m) 35s ago 24m 50.5M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:44:25.943 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (24m) 35s ago 24m 55.1M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:44:25.943 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (86s) 35s ago 23m 54.0M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:44:25.943 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (22m) 58s ago 22m 75.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:44:25.943 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (22m) 35s ago 22m 75.5M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:44:26.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:26 smithi070 ceph-mon[21827]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-04-24T16:44:26.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:26 smithi070 ceph-mon[26293]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-04-24T16:44:26.348 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:44:26.349 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:44:26.350 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-24T16:44:26.350 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:44:26.350 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:44:26.350 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:44:26.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:26 smithi156 ceph-mon[22679]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-04-24T16:44:26.683 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:44:26.683 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:44:26.684 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:44:26.684 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi156", 2024-04-24T16:44:26.684 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:44:26.684 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "0/1 daemons upgraded", 2024-04-24T16:44:26.684 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-04-24T16:44:26.684 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:44:26.684 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:44:26.684 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:44:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:44:26] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.43.0" 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='client.15729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:27.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: Upgrade: Setting container_image for all mgr 2024-04-24T16:44:27.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:27.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:44:27.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-24T16:44:27.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: Upgrade: It appears safe to stop mon.b 2024-04-24T16:44:27.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:27 smithi156 ceph-mon[22679]: from='client.? 172.21.15.70:0/120773476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:27.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='client.15729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all mgr 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: Upgrade: It appears safe to stop mon.b 2024-04-24T16:44:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/120773476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='client.15729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:27.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:27.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all mgr 2024-04-24T16:44:27.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:27.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:44:27.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-24T16:44:27.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: Upgrade: It appears safe to stop mon.b 2024-04-24T16:44:27.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:27 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/120773476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:28.302 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:28 smithi156 ceph-mon[22679]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:28.302 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:28 smithi156 ceph-mon[22679]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:28.302 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:28 smithi156 ceph-mon[22679]: from='client.25636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:28.302 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:28 smithi156 ceph-mon[22679]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:28.303 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:28 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:44:28.303 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:28 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:44:28.303 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:28 smithi156 ceph-mon[22679]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[26293]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[26293]: from='client.25636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:44:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:44:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:28.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:28.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[21827]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:28.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[21827]: from='client.25636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:28.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:28.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:44:28.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:44:28.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:28 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:29.294 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:29 smithi156 ceph-mon[22679]: Upgrade: Updating mon.b 2024-04-24T16:44:29.294 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:29 smithi156 ceph-mon[22679]: Deploying daemon mon.b on smithi156 2024-04-24T16:44:29.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:29 smithi070 ceph-mon[26293]: Upgrade: Updating mon.b 2024-04-24T16:44:29.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:29 smithi070 ceph-mon[26293]: Deploying daemon mon.b on smithi156 2024-04-24T16:44:29.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:29 smithi070 ceph-mon[21827]: Upgrade: Updating mon.b 2024-04-24T16:44:29.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:29 smithi070 ceph-mon[21827]: Deploying daemon mon.b on smithi156 2024-04-24T16:44:29.852 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:29 smithi156 systemd[1]: Stopping Ceph mon.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:44:29.852 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:29 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b[22673]: 2024-04-24T16:44:29.789+0000 7fd3d5569700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:44:29.852 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:29 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b[22673]: 2024-04-24T16:44:29.789+0000 7fd3d5569700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-24T16:44:30.150 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:29 smithi156 podman[64933]: 2024-04-24 16:44:29.85287182 +0000 UTC m=+0.158237582 container died 9f6c28bccd1d0d19347638e046f9735cf19ef608a9eab9be58a694ed98ba8cf4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-04-24T16:44:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:30 smithi156 podman[64933]: 2024-04-24 16:44:30.249346292 +0000 UTC m=+0.554712063 container cleanup 9f6c28bccd1d0d19347638e046f9735cf19ef608a9eab9be58a694ed98ba8cf4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, ceph=True) 2024-04-24T16:44:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:30 smithi156 bash[64933]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b 2024-04-24T16:44:30.867 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:30 smithi156 podman[64947]: 2024-04-24 16:44:30.442387569 +0000 UTC m=+0.592185861 container remove 9f6c28bccd1d0d19347638e046f9735cf19ef608a9eab9be58a694ed98ba8cf4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , version=8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, name=centos-stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-24T16:44:30.867 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:30 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-24T16:44:30.867 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:30 smithi156 systemd[1]: Stopped Ceph mon.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:44:30.867 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:30 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.b.service: Consumed 18.065s CPU time. 2024-04-24T16:44:30.867 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:30 smithi156 systemd[1]: Starting Ceph mon.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:44:31.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 podman[65078]: 2024-04-24 16:44:30.987541707 +0000 UTC m=+0.018548240 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:44:31.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 podman[65078]: 2024-04-24 16:44:31.101743651 +0000 UTC m=+0.132750180 container create 421c34aad07fdc440ee676357a13929e4ccf510fec6c416dd08357eec992853e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:44:31.593 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 podman[65078]: 2024-04-24 16:44:31.338197351 +0000 UTC m=+0.369203884 container init 421c34aad07fdc440ee676357a13929e4ccf510fec6c416dd08357eec992853e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424) 2024-04-24T16:44:31.593 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 podman[65078]: 2024-04-24 16:44:31.341693849 +0000 UTC m=+0.372700379 container start 421c34aad07fdc440ee676357a13929e4ccf510fec6c416dd08357eec992853e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-24T16:44:31.593 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T16:44:31.593 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev), process ceph-mon, pid 2 2024-04-24T16:44:31.593 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: pidfile_write: ignore empty --pid-file 2024-04-24T16:44:31.593 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: load: jerasure load: lrc 2024-04-24T16:44:31.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: RocksDB version: 7.9.2 2024-04-24T16:44:31.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Git sha 0 2024-04-24T16:44:31.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Compile date 2024-04-24 13:56:12 2024-04-24T16:44:31.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: DB SUMMARY 2024-04-24T16:44:31.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: DB Session ID: 19IZUMA3H5LOGRKJIJZI 2024-04-24T16:44:31.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: CURRENT file: CURRENT 2024-04-24T16:44:31.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T16:44:31.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2851 Bytes 2024-04-24T16:44:31.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-04-24T16:44:31.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2716547 ; 2024-04-24T16:44:31.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.error_if_exists: 0 2024-04-24T16:44:31.595 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.create_if_missing: 0 2024-04-24T16:44:31.596 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.paranoid_checks: 1 2024-04-24T16:44:31.596 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T16:44:31.596 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T16:44:31.596 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T16:44:31.596 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.env: 0x555901164cc0 2024-04-24T16:44:31.596 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T16:44:31.597 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.info_log: 0x555901e71d80 2024-04-24T16:44:31.597 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T16:44:31.599 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.statistics: (nil) 2024-04-24T16:44:31.599 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.use_fsync: 0 2024-04-24T16:44:31.599 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_log_file_size: 0 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.allow_fallocate: 1 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T16:44:31.600 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.use_direct_reads: 0 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.db_log_dir: 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.wal_dir: 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T16:44:31.601 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.write_buffer_manager: 0x555901e43220 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T16:44:31.602 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.unordered_write: 0 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T16:44:31.603 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.row_cache: None 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.wal_filter: None 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.two_write_queues: 0 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.wal_compression: 0 2024-04-24T16:44:31.604 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.atomic_flush: 0 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.log_readahead_size: 0 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T16:44:31.605 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_background_jobs: 2 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_background_compactions: -1 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_subcompactions: 1 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T16:44:31.606 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_open_files: -1 2024-04-24T16:44:31.607 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_background_flushes: -1 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Compression algorithms supported: 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kZSTD supported: 0 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kXpressCompression supported: 0 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T16:44:31.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kZlibCompression supported: 1 2024-04-24T16:44:31.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kSnappyCompression supported: 1 2024-04-24T16:44:31.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kLZ4Compression supported: 1 2024-04-24T16:44:31.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: kBZip2Compression supported: 0 2024-04-24T16:44:31.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T16:44:31.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T16:44:31.609 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-04-24T16:44:31.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T16:44:31.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T16:44:31.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.merge_operator: 2024-04-24T16:44:31.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_filter: None 2024-04-24T16:44:31.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T16:44:31.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T16:44:31.610 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555901e71f40) 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks: 1 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout: pin_top_level_index_and_filter: 1 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_type: 0 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_index_type: 0 2024-04-24T16:44:31.611 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_shortening: 1 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: checksum: 4 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: no_block_cache: 0 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache: 0x555901e751f0 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_name: BinnedLRUCache 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_options: 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: capacity : 536870912 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: num_shard_bits : 4 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: strict_capacity_limit : 0 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: high_pri_pool_ratio: 0.000 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_cache_compressed: (nil) 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: persistent_cache: (nil) 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size: 4096 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_size_deviation: 10 2024-04-24T16:44:31.612 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_restart_interval: 16 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: index_block_restart_interval: 1 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: metadata_block_size: 4096 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: partition_filters: 0 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: use_delta_encoding: 1 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: filter_policy: bloomfilter 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: whole_key_filtering: 1 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: verify_compression: 0 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: read_amp_bytes_per_bit: 0 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: format_version: 5 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_index_compression: 1 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: block_align: 0 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: max_auto_readahead_size: 262144 2024-04-24T16:44:31.613 INFO:journalctl@ceph.mon.b.smithi156.stdout: prepopulate_block_cache: 0 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout: initial_auto_readahead_size: 8192 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression: NoCompression 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.num_levels: 7 2024-04-24T16:44:31.614 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T16:44:31.615 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T16:44:31.616 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T16:44:31.617 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T16:44:31.618 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.inplace_update_support: 0 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T16:44:31.619 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.bloom_locality: 0 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.max_successive_merges: 0 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T16:44:31.620 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.ttl: 2592000 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.enable_blob_files: false 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.min_blob_size: 0 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T16:44:31.621 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28887, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d674013e-90cc-4f1c-a0bd-31520bd90ae9 2024-04-24T16:44:31.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977071371321, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977071378552, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1653937, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28893, "largest_seqno": 30529, "table_properties": {"data_size": 1647690, "index_size": 3542, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 15779, "raw_average_key_size": 24, "raw_value_size": 1634324, "raw_average_value_size": 2537, "num_data_blocks": 160, "num_entries": 644, "num_filter_entries": 644, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713977071, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d674013e-90cc-4f1c-a0bd-31520bd90ae9", "db_session_id": "19IZUMA3H5LOGRKJIJZI", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977071378650, "job": 1, "event": "recovery_finished"} 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555901f92000 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: DB pointer 0x555901f7e000 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** DB Stats ** 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:44:31.623 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T16:44:31.624 INFO:journalctl@ceph.mon.b.smithi156.stdout: L0 1/0 1.58 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 485.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:44:31.625 INFO:journalctl@ceph.mon.b.smithi156.stdout: L6 1/0 9.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-24T16:44:31.625 INFO:journalctl@ceph.mon.b.smithi156.stdout: Sum 2/0 11.17 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 485.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:44:31.625 INFO:journalctl@ceph.mon.b.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 485.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:44:31.625 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:44:31.625 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** Compaction Stats [default] ** 2024-04-24T16:44:31.625 INFO:journalctl@ceph.mon.b.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T16:44:31.626 INFO:journalctl@ceph.mon.b.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:44:31.626 INFO:journalctl@ceph.mon.b.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 485.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:44:31.626 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:44:31.626 INFO:journalctl@ceph.mon.b.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T16:44:31.626 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:44:31.626 INFO:journalctl@ceph.mon.b.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: Cumulative compaction: 0.00 GB write, 164.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: Interval compaction: 0.00 GB write, 164.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: Block cache BinnedLRUCache@0x555901e751f0#2 capacity: 512.00 MB usage: 5.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: starting mon.b rank 2 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:44:31.627 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???) e3 preinit fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???).mds e1 new map 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???).mds e1 print_map 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout: e1 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout: legacy client fscid: -1 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout: 2024-04-24T16:44:31.628 INFO:journalctl@ceph.mon.b.smithi156.stdout: No filesystems configured 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???).osd e111 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 bash[65078]: 421c34aad07fdc440ee676357a13929e4ccf510fec6c416dd08357eec992853e 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 systemd[1]: Started Ceph mon.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:44:31.629 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:31 smithi156 ceph-mon[65097]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:32.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:31 smithi070 ceph-mon[26293]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:32.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:31 smithi070 ceph-mon[21827]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:32.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-04-24T16:44:32.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: mon.b calling monitor election 2024-04-24T16:44:32.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: mon.c calling monitor election 2024-04-24T16:44:32.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: mon.a calling monitor election 2024-04-24T16:44:32.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:44:32.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:44:32.729 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: fsmap 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: mgrmap e43: y(active, since 2m), standbys: x 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:32.730 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:32 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-04-24T16:44:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: mon.b calling monitor election 2024-04-24T16:44:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: mon.c calling monitor election 2024-04-24T16:44:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: mon.a calling monitor election 2024-04-24T16:44:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: fsmap 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: mgrmap e43: y(active, since 2m), standbys: x 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-04-24T16:44:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: mon.b calling monitor election 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: mon.c calling monitor election 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: mon.a calling monitor election 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: monmap e3: 3 mons at {a=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],b=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0],c=[v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0]} 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: fsmap 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: mgrmap e43: y(active, since 2m), standbys: x 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:33.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:32 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:33 smithi156 ceph-mon[65097]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:34.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:33 smithi070 ceph-mon[26293]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:34.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:33 smithi070 ceph-mon[21827]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:36.275 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:35 smithi070 ceph-mon[26293]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:36.276 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:35 smithi070 ceph-mon[21827]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:36.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:35 smithi156 ceph-mon[65097]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:36.542 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:44:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:44:36.386+0000 7ff4ba899700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T16:44:36.543 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:44:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:44:36] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.43.0" 2024-04-24T16:44:37.409 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:37 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:37 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:37.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:37 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:37.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:37 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:37.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:37 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:37.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:37 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:38 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:38 smithi156 ceph-mon[65097]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:38 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:38 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[21827]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:38.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:38.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[26293]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:38.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:38.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:38 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:40.342 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:40 smithi156 ceph-mon[65097]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:40.342 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:40 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:40 smithi070 ceph-mon[21827]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:40 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:40 smithi070 ceph-mon[26293]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:40 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:42.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:42.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Detected new or changed devices on smithi156 2024-04-24T16:44:42.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all crash 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all mds 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all nfs 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi156 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:44:42.046 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all crash 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all mds 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:44:42.047 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all nfs 2024-04-24T16:44:42.048 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all node-exporter 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.049 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all prometheus 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all alertmanager 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all grafana 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all loki 2024-04-24T16:44:42.050 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all promtail 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Finalizing container_image settings 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:44:42.051 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all node-exporter 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all prometheus 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all alertmanager 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.052 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all grafana 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all loki 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all promtail 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Finalizing container_image settings 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:44:42.053 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:44:42.054 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:44:42.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:44:42.057 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.058 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: Upgrade: Complete! 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:44:42.059 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.060 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: Upgrade: Complete! 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.061 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 7 op/s 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-24T16:44:42.062 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 7 op/s 2024-04-24T16:44:42.063 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:41 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Detected new or changed devices on smithi156 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all crash 2024-04-24T16:44:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mds 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nfs 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all node-exporter 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all prometheus 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all alertmanager 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all grafana 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all loki 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all promtail 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Finalizing container_image settings 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:44:42.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:44:42.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:44:42.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:44:42.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:44:42.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:44:42.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: Upgrade: Complete! 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.169 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 7 op/s 2024-04-24T16:44:42.170 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:41 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:44.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:43 smithi070 ceph-mon[26293]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:44.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:43 smithi070 ceph-mon[21827]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:43 smithi156 ceph-mon[65097]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:45 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:45 smithi156 ceph-mon[65097]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:46.280 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:45 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:46.280 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:45 smithi070 ceph-mon[21827]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:46.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:45 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:44:46.280 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:45 smithi070 ceph-mon[26293]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:44:46.542 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:44:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:44:46] "GET /metrics HTTP/1.1" 200 38007 "" "Prometheus/2.43.0" 2024-04-24T16:44:48.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:48 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:48.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:48 smithi070 ceph-mon[26293]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-04-24T16:44:48.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:48 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:48.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:48 smithi070 ceph-mon[21827]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-04-24T16:44:48.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:48 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:48.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:48 smithi156 ceph-mon[65097]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-04-24T16:44:50.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:50 smithi070 ceph-mon[21827]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-04-24T16:44:50.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:50 smithi070 ceph-mon[26293]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-04-24T16:44:50.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:50 smithi156 ceph-mon[65097]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-04-24T16:44:52.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:52 smithi070 ceph-mon[26293]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:52.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:52 smithi070 ceph-mon[21827]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:52.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:52 smithi156 ceph-mon[65097]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:44:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:53 smithi070 ceph-mon[26293]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:53 smithi070 ceph-mon[21827]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:53 smithi156 ceph-mon[65097]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:55.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:54 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:55.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:54 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:54 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:44:56.287 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:55 smithi070 ceph-mon[26293]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:56.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:55 smithi070 ceph-mon[21827]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:55 smithi156 ceph-mon[65097]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:44:56.542 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:44:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:44:56] "GET /metrics HTTP/1.1" 200 38009 "" "Prometheus/2.43.0" 2024-04-24T16:44:57.770 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:44:58.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:58 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:58.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:58 smithi070 ceph-mon[26293]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:58.392 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:44:58 smithi070 ceph-mon[26293]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:58.392 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:58 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:58.392 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:58 smithi070 ceph-mon[21827]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:58.392 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:44:58 smithi070 ceph-mon[21827]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:58.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:58 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:44:58.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:58 smithi156 ceph-mon[65097]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:44:58.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:44:58 smithi156 ceph-mon[65097]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:44:59.833 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:44:59.833 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (97s) 92s ago 23m 16.7M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:44:59.833 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (74s) 23s ago 23m 76.4M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:44:59.833 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 unknown 92s ago 22m 58.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:44:59.834 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (2m) 23s ago 28m 452M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:44:59.834 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (12m) 92s ago 29m 503M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:44:59.834 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (29m) 92s ago 29m 78.1M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:44:59.834 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (28s) 23s ago 28m 19.7M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:44:59.834 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (28m) 92s ago 28m 56.3M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:44:59.834 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (2m) 92s ago 24m 8568k - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:44:59.835 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (2m) 23s ago 24m 15.6M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:44:59.835 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (27m) 92s ago 27m 57.0M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:44:59.835 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (27m) 92s ago 27m 58.1M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:44:59.835 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (26m) 92s ago 26m 50.9M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:44:59.835 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (26m) 92s ago 26m 52.2M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:44:59.835 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (26m) 23s ago 26m 54.7M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:44:59.836 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (25m) 23s ago 25m 53.2M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:44:59.836 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (25m) 23s ago 25m 50.6M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:44:59.836 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (25m) 23s ago 25m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:44:59.836 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (2m) 23s ago 24m 62.1M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:44:59.836 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (22m) 92s ago 22m 75.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:44:59.836 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (22m) 23s ago 22m 75.8M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:45:00.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:00 smithi070 ceph-mon[21827]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:00.487 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:00 smithi070 ceph-mon[26293]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:00.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-04-24T16:45:00.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:00 smithi156 ceph-mon[65097]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:01.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:01 smithi070 ceph-mon[26293]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:01.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:01 smithi070 ceph-mon[21827]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:01.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:01 smithi156 ceph-mon[65097]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:02.290 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:02 smithi070 ceph-mon[21827]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:02.290 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:02 smithi070 ceph-mon[26293]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:02 smithi156 ceph-mon[65097]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:02.728 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:45:03.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:03 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/909637223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:03.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:03 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/909637223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:03.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:45:03.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:03 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/909637223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:04.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:04 smithi070 ceph-mon[26293]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:04.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:04 smithi070 ceph-mon[21827]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:04.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:04 smithi156 ceph-mon[65097]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": null, 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": false, 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "", 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": null, 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:45:05.407 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:45:06.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:45:06.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:06 smithi070 ceph-mon[21827]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:06.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:06 smithi070 ceph-mon[26293]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:06.554 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:45:06] "GET /metrics HTTP/1.1" 200 38009 "" "Prometheus/2.43.0" 2024-04-24T16:45:06.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:06 smithi156 ceph-mon[65097]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:07 smithi070 ceph-mon[26293]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:07 smithi070 ceph-mon[21827]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:07.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:07 smithi156 ceph-mon[65097]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:08.109 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:08.109 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:45:08.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:08 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:08.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:08 smithi070 ceph-mon[26293]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:08.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:08 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/4005916142' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:45:08.424 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:08 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:08.424 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:08 smithi070 ceph-mon[21827]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:08.424 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:08 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/4005916142' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:45:08.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:08 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:08.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:08 smithi156 ceph-mon[65097]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:08.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:08 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/4005916142' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:45:08.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-04-24T16:45:10.260 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:10 smithi070 ceph-mon[21827]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:10.260 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:10 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:10.260 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:10 smithi070 ceph-mon[26293]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:10.261 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:10 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:10.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:10 smithi156 ceph-mon[65097]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:10.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:10 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:12.276 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:12 smithi070 ceph-mon[26293]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:12.276 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:12 smithi070 ceph-mon[26293]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mon", "hosts": "smithi070", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:12.276 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:12 smithi070 ceph-mon[26293]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:12.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:12 smithi070 ceph-mon[21827]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:12.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:12 smithi070 ceph-mon[21827]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mon", "hosts": "smithi070", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:12.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:12 smithi070 ceph-mon[21827]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:12.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:12 smithi156 ceph-mon[65097]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:12.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:12 smithi156 ceph-mon[65097]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "mon", "hosts": "smithi070", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:12.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:12 smithi156 ceph-mon[65097]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:13.142 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:45:13.948 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-24T16:45:14.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:14.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:45:14.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:14.284 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:14.285 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:14.286 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-04-24T16:45:14.286 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:14.286 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:14 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:45:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-04-24T16:45:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:14 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:15 smithi156 ceph-mon[65097]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:45:15.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:15 smithi070 ceph-mon[26293]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:45:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:15 smithi070 ceph-mon[21827]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:45:16.303 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:45:16.400 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:16 smithi070 ceph-mon[26293]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 635 B/s rd, 0 op/s 2024-04-24T16:45:16.400 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:16 smithi070 ceph-mon[21827]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 635 B/s rd, 0 op/s 2024-04-24T16:45:16.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:16 smithi156 ceph-mon[65097]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 635 B/s rd, 0 op/s 2024-04-24T16:45:16.746 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:45:16] "GET /metrics HTTP/1.1" 200 38008 "" "Prometheus/2.43.0" 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (114s) 109s ago 23m 16.7M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (91s) 40s ago 23m 76.4M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 unknown 109s ago 23m 58.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (3m) 40s ago 28m 452M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (13m) 109s ago 29m 503M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (29m) 109s ago 29m 78.1M 2048M 17.2.0 e1d6a67b021e d744477c6ebb 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (46s) 40s ago 28m 19.7M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:45:17.015 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (28m) 109s ago 28m 56.3M 2048M 17.2.0 e1d6a67b021e 2b11cd568406 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (2m) 109s ago 24m 8568k - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (2m) 40s ago 24m 15.6M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (28m) 109s ago 28m 57.0M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (27m) 109s ago 27m 58.1M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (27m) 109s ago 27m 50.9M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (26m) 109s ago 26m 52.2M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (26m) 40s ago 26m 54.7M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (26m) 40s ago 26m 53.2M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (25m) 40s ago 25m 50.6M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (25m) 40s ago 25m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (2m) 40s ago 24m 62.1M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (23m) 109s ago 23m 75.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:45:17.016 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (23m) 40s ago 23m 75.8M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:45:17.436 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:45:17.436 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:45:17.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-24T16:45:17.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 1 2024-04-24T16:45:17.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:17.436 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:45:17.436 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": {}, 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:45:17.437 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi070", 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "0/2 daemons upgraded", 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-04-24T16:45:17.783 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:45:17.784 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='client.25690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:45:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: Upgrade: Setting container_image for all mgr 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: Upgrade: It appears safe to stop mon.a 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[26293]: from='client.? 172.21.15.70:0/937396266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:18.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='client.25690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: Upgrade: Setting container_image for all mgr 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-24T16:45:18.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: Upgrade: It appears safe to stop mon.a 2024-04-24T16:45:18.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:18.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:18 smithi070 ceph-mon[21827]: from='client.? 172.21.15.70:0/937396266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='client.25690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mgr 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: Upgrade: It appears safe to stop mon.a 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:18.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:18 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/937396266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:19 smithi156 ceph-mon[65097]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:19 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:19 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:19 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:19 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:19.474 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[26293]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[21827]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[21827]: from='mgr.25297 ' entity='mgr.y' 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:19.475 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:19.476 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:19 smithi070 ceph-mon[21827]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:20.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-mon[26293]: Upgrade: Updating mon.a 2024-04-24T16:45:20.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-mon[26293]: Deploying daemon mon.a on smithi070 2024-04-24T16:45:20.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-mon[26293]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 635 B/s rd, 0 op/s 2024-04-24T16:45:20.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-mon[21827]: Upgrade: Updating mon.a 2024-04-24T16:45:20.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-mon[21827]: Deploying daemon mon.a on smithi070 2024-04-24T16:45:20.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-mon[21827]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 635 B/s rd, 0 op/s 2024-04-24T16:45:20.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 systemd[1]: Stopping Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:45:20.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:20 smithi156 ceph-mon[65097]: Upgrade: Updating mon.a 2024-04-24T16:45:20.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:20 smithi156 ceph-mon[65097]: Deploying daemon mon.a on smithi070 2024-04-24T16:45:20.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:20 smithi156 ceph-mon[65097]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 635 B/s rd, 0 op/s 2024-04-24T16:45:20.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a[21823]: 2024-04-24T16:45:20.576+0000 7fd2b49ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:45:20.607 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a[21823]: 2024-04-24T16:45:20.576+0000 7fd2b49ca700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-24T16:45:20.902 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 podman[81437]: 2024-04-24 16:45:20.607568328 +0000 UTC m=+0.113862653 container died d744477c6ebbebcf8d9b22a9231138e8865b3e2e902bd32dd42a577bfc068c2c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, architecture=x86_64, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0) 2024-04-24T16:45:21.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 podman[81437]: 2024-04-24 16:45:20.993725391 +0000 UTC m=+0.500019715 container cleanup d744477c6ebbebcf8d9b22a9231138e8865b3e2e902bd32dd42a577bfc068c2c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, distribution-scope=public, ceph=True, release=754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, version=8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.display-name=CentOS Stream 8) 2024-04-24T16:45:21.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:20 smithi070 bash[81437]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a 2024-04-24T16:45:21.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 podman[81451]: 2024-04-24 16:45:21.228227769 +0000 UTC m=+0.614136227 container remove d744477c6ebbebcf8d9b22a9231138e8865b3e2e902bd32dd42a577bfc068c2c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.19.8, io.openshift.expose-services=, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, vendor=Red Hat, Inc., name=centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-04-24T16:45:21.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-24T16:45:21.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 systemd[1]: Stopped Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:45:21.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service: Consumed 25.647s CPU time. 2024-04-24T16:45:21.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 systemd[1]: Starting Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:45:21.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 podman[81583]: 2024-04-24 16:45:21.788054089 +0000 UTC m=+0.115067221 container create 538ac8b89d6a37ce41dc5e1c58a8e1025fc86c81012411a7c202e4f50c4b8dfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:45:21.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 podman[81583]: 2024-04-24 16:45:21.692303001 +0000 UTC m=+0.019316152 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 podman[81583]: 2024-04-24 16:45:21.982105328 +0000 UTC m=+0.309118460 container init 538ac8b89d6a37ce41dc5e1c58a8e1025fc86c81012411a7c202e4f50c4b8dfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:21 smithi070 podman[81583]: 2024-04-24 16:45:21.985242722 +0000 UTC m=+0.312255872 container start 538ac8b89d6a37ce41dc5e1c58a8e1025fc86c81012411a7c202e4f50c4b8dfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev), process ceph-mon, pid 2 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: pidfile_write: ignore empty --pid-file 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: load: jerasure load: lrc 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: RocksDB version: 7.9.2 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Git sha 0 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Compile date 2024-04-24 13:56:12 2024-04-24T16:45:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: DB SUMMARY 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: DB Session ID: 3PZ15N40R4MDK3ATTS06 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: CURRENT file: CURRENT 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: MANIFEST file: MANIFEST-000015 size: 3117 Bytes 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000063.sst 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000061.log size: 174760 ; 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.error_if_exists: 0 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.create_if_missing: 0 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.paranoid_checks: 1 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.env: 0x56394fd25cc0 2024-04-24T16:45:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T16:45:22.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.info_log: 0x5639508add80 2024-04-24T16:45:22.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.statistics: (nil) 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.use_fsync: 0 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_log_file_size: 0 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T16:45:22.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.allow_fallocate: 1 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.use_direct_reads: 0 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.db_log_dir: 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.wal_dir: 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T16:45:22.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.write_buffer_manager: 0x56395087f220 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.unordered_write: 0 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T16:45:22.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.row_cache: None 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.wal_filter: None 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.two_write_queues: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.wal_compression: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.atomic_flush: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.log_readahead_size: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T16:45:22.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_background_jobs: 2 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_background_compactions: -1 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_subcompactions: 1 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T16:45:22.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_open_files: -1 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_background_flushes: -1 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Compression algorithms supported: 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kZSTD supported: 0 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kXpressCompression supported: 0 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kZlibCompression supported: 1 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kSnappyCompression supported: 1 2024-04-24T16:45:22.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kLZ4Compression supported: 1 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: kBZip2Compression supported: 0 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.merge_operator: 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_filter: None 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5639508adf40) 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-04-24T16:45:22.302 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_type: 0 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_index_type: 0 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_shortening: 1 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: checksum: 4 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: no_block_cache: 0 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache: 0x5639508b11f0 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_options: 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: capacity : 536870912 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_shard_bits : 4 2024-04-24T16:45:22.303 INFO:journalctl@ceph.mon.a.smithi070.stdout: strict_capacity_limit : 0 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_compressed: (nil) 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: persistent_cache: (nil) 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size: 4096 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size_deviation: 10 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_restart_interval: 16 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_block_restart_interval: 1 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: metadata_block_size: 4096 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: partition_filters: 0 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_delta_encoding: 1 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: filter_policy: bloomfilter 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: whole_key_filtering: 1 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: verify_compression: 0 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-04-24T16:45:22.304 INFO:journalctl@ceph.mon.a.smithi070.stdout: format_version: 5 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_index_compression: 1 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_align: 0 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_auto_readahead_size: 262144 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout: prepopulate_block_cache: 0 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout: initial_auto_readahead_size: 8192 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression: NoCompression 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.num_levels: 7 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T16:45:22.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T16:45:22.306 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T16:45:22.307 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T16:45:22.308 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.inplace_update_support: 0 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T16:45:22.309 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.bloom_locality: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.max_successive_merges: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.ttl: 2592000 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.enable_blob_files: false 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.min_blob_size: 0 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T16:45:22.310 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 63.sst 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 65, last_sequence is 29317, log_number is 61,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 61 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3c4bd05e-f157-4081-8681-49ef00aebfdb 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977122015395, "job": 1, "event": "recovery_started", "wal_files": [61]} 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #61 mode 2 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977122016714, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 66, "file_size": 171150, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 29318, "largest_seqno": 29517, "table_properties": {"data_size": 169378, "index_size": 414, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2688, "raw_average_key_size": 25, "raw_value_size": 167173, "raw_average_value_size": 1562, "num_data_blocks": 19, "num_entries": 107, "num_filter_entries": 107, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713977122, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3c4bd05e-f157-4081-8681-49ef00aebfdb", "db_session_id": "3PZ15N40R4MDK3ATTS06", "orig_file_number": 66, "seqno_to_time_mapping": "N/A"}} 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977122016796, "job": 1, "event": "recovery_finished"} 2024-04-24T16:45:22.311 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/version_set.cc:5047] Creating manifest 68 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000061.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5639509ce000 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: DB pointer 0x5639509ba000 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** DB Stats ** 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:45:22.312 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: L0 1/0 167.14 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: L6 1/0 10.04 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: Sum 2/0 10.21 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 175.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T16:45:22.313 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative compaction: 0.00 GB write, 47.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval compaction: 0.00 GB write, 47.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache BinnedLRUCache@0x5639508b11f0#2 capacity: 512.00 MB usage: 0.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.48 KB,9.23872e-05%) Misc(1,0.00 KB,0%) 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:45:22.314 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: starting mon.a rank 0 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???) e3 preinit fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???).mds e1 new map 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???).mds e1 print_map 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout: e1 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout: legacy client fscid: -1 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout: No filesystems configured 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???).osd e111 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:45:22.315 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 ceph-mon[81602]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-04-24T16:45:22.316 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 bash[81583]: 538ac8b89d6a37ce41dc5e1c58a8e1025fc86c81012411a7c202e4f50c4b8dfb 2024-04-24T16:45:22.316 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:22 smithi070 systemd[1]: Started Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:45:25.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: ignoring --setuser ceph since I am not root 2024-04-24T16:45:25.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: ignoring --setgroup ceph since I am not root 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mgr[58027]: -- 172.21.15.156:0/3434964410 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602573e65a0 con 0x5602573bb000 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:25.232+0000 7f48386ae200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: mon.a calling monitor election 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: monmap epoch 3 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: last_changed 2024-04-24T16:16:44.272729+0000 2024-04-24T16:45:25.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: created 2024-04-24T16:15:12.900488+0000 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: min_mon_release 17 (quincy) 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: election_strategy: 1 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: 1: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: fsmap 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: mgrmap e43: y(active, since 3m), standbys: x 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: from='mgr.25297 ' entity='' 2024-04-24T16:45:25.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-mon[65097]: mgrmap e44: y(active, since 3m), standbys: x 2024-04-24T16:45:25.424 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ignoring --setuser ceph since I am not root 2024-04-24T16:45:25.424 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ignoring --setgroup ceph since I am not root 2024-04-24T16:45:25.424 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mgr[60576]: -- 172.21.15.70:0/673176319 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c8bf97c5a0 con 0x55c8bf951000 2024-04-24T16:45:25.424 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:25.238+0000 7f22c5bd1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T16:45:25.424 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:25.423+0000 7f22c5bd1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:45:25.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:25.424 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: mon.a calling monitor election 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: monmap epoch 3 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: last_changed 2024-04-24T16:16:44.272729+0000 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: created 2024-04-24T16:15:12.900488+0000 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: min_mon_release 17 (quincy) 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: election_strategy: 1 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: 1: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:25.425 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: fsmap 2024-04-24T16:45:25.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:45:25.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: mgrmap e43: y(active, since 3m), standbys: x 2024-04-24T16:45:25.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:25.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:25.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:45:25.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: from='mgr.25297 ' entity='' 2024-04-24T16:45:25.426 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[26293]: mgrmap e44: y(active, since 3m), standbys: x 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: from='mgr.25297 172.21.15.70:0/3599466351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: mon.a calling monitor election 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: monmap epoch 3 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: last_changed 2024-04-24T16:16:44.272729+0000 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: created 2024-04-24T16:15:12.900488+0000 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: min_mon_release 17 (quincy) 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: election_strategy: 1 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-24T16:45:25.427 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: 1: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: fsmap 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: osdmap e111: 8 total, 8 up, 8 in 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: mgrmap e43: y(active, since 3m), standbys: x 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: from='mgr.25297 ' entity='' 2024-04-24T16:45:25.428 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-mon[81602]: mgrmap e44: y(active, since 3m), standbys: x 2024-04-24T16:45:25.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:25.525+0000 7f22c5bd1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:45:25.829 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:25.419+0000 7f48386ae200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:45:25.829 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:25.521+0000 7f48386ae200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T16:45:25.829 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:25.829+0000 7f48386ae200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:45:26.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:25.915+0000 7f48386ae200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:45:26.292 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:25.830+0000 7f22c5bd1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T16:45:26.293 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:25 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:25.915+0000 7f22c5bd1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T16:45:27.301 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:27 smithi156 ceph-mon[65097]: mgrmap e45: y(active, since 3m), standbys: x 2024-04-24T16:45:27.614 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:27.128+0000 7f48386ae200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:45:27.614 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:27.301+0000 7f48386ae200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:45:27.614 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:27.474+0000 7f48386ae200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:45:27.614 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:27 smithi070 ceph-mon[26293]: mgrmap e45: y(active, since 3m), standbys: x 2024-04-24T16:45:27.615 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:27 smithi070 ceph-mon[81602]: mgrmap e45: y(active, since 3m), standbys: x 2024-04-24T16:45:27.615 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:27.120+0000 7f22c5bd1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T16:45:27.615 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:27.292+0000 7f22c5bd1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T16:45:27.615 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:27.464+0000 7f22c5bd1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T16:45:27.891 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:27.795+0000 7f48386ae200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:45:27.891 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:27.891+0000 7f48386ae200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:45:28.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:27.753+0000 7f22c5bd1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T16:45:28.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:27.847+0000 7f22c5bd1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T16:45:28.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:28.099+0000 7f48386ae200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:45:28.543 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:28.053+0000 7f22c5bd1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T16:45:28.543 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:28.250+0000 7f22c5bd1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:45:28.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:28.296+0000 7f48386ae200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T16:45:29.160 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:28 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:28.901+0000 7f48386ae200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:45:29.284 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:28 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:28.849+0000 7f22c5bd1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T16:45:29.542 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:29.284+0000 7f22c5bd1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:45:29.543 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:29.384+0000 7f22c5bd1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:45:29.642 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:29 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:29.333+0000 7f48386ae200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T16:45:29.642 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:29 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:29.432+0000 7f48386ae200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T16:45:29.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:29 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:29.640+0000 7f48386ae200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:45:30.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:29.590+0000 7f22c5bd1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T16:45:30.461 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:30.118+0000 7f22c5bd1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:45:30.461 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:30.213+0000 7f22c5bd1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:45:30.523 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:30.176+0000 7f48386ae200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T16:45:30.523 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:30.271+0000 7f48386ae200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T16:45:30.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:30.461+0000 7f22c5bd1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:45:30.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:30.597+0000 7f22c5bd1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:45:30.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:30.695+0000 7f22c5bd1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:45:30.875 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:30.522+0000 7f48386ae200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T16:45:30.875 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:30.658+0000 7f48386ae200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T16:45:30.876 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:30.755+0000 7f48386ae200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T16:45:30.876 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:30.874+0000 7f48386ae200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:45:31.118 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:30 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:30.814+0000 7f22c5bd1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T16:45:31.118 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:31.118+0000 7f22c5bd1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:45:31.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:31.182+0000 7f48386ae200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T16:45:32.013 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:31.732+0000 7f22c5bd1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:45:32.013 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:31.959+0000 7f22c5bd1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:45:32.099 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:31.803+0000 7f48386ae200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:32.032+0000 7f48386ae200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: [24/Apr/2024:16:45:32] ENGINE Bus STARTING 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: Active manager daemon y restarted 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: Activating manager daemon y 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: mgrmap e46: y(active, starting, since 0.00761489s), standbys: x 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:45:32.100 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:45:32.101 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:45:32.102 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-mon[65097]: Manager daemon y is now available 2024-04-24T16:45:32.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: Active manager daemon y restarted 2024-04-24T16:45:32.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: Activating manager daemon y 2024-04-24T16:45:32.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:32.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: mgrmap e46: y(active, starting, since 0.00761489s), standbys: x 2024-04-24T16:45:32.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:45:32.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:45:32.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:45:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[81602]: Manager daemon y is now available 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: Active manager daemon y restarted 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: Activating manager daemon y 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: mgrmap e46: y(active, starting, since 0.00761489s), standbys: x 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:45:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T16:45:32.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-mon[26293]: Manager daemon y is now available 2024-04-24T16:45:32.297 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:45:32] ENGINE Bus STARTING 2024-04-24T16:45:32.297 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: CherryPy Checker: 2024-04-24T16:45:32.297 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: The Application mounted at '' has an empty config. 2024-04-24T16:45:32.297 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:32.297 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:45:32] ENGINE Serving on http://:::9283 2024-04-24T16:45:32.297 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:45:32] ENGINE Bus STARTED 2024-04-24T16:45:32.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: CherryPy Checker: 2024-04-24T16:45:32.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: The Application mounted at '' has an empty config. 2024-04-24T16:45:32.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: 2024-04-24T16:45:32.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: [24/Apr/2024:16:45:32] ENGINE Serving on http://:::9283 2024-04-24T16:45:32.410 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:45:32 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x[58023]: [24/Apr/2024:16:45:32] ENGINE Bus STARTED 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: Standby manager daemon x restarted 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: Standby manager daemon x started 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:45:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[81602]: mgrmap e47: y(active, since 1.00998s), standbys: x 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:32.973+0000 7f22938a6700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: Standby manager daemon x restarted 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: Standby manager daemon x started 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:45:33.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:45:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:45:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:45:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:45:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:45:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:33 smithi070 ceph-mon[26293]: mgrmap e47: y(active, since 1.00998s), standbys: x 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: Standby manager daemon x restarted 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: Standby manager daemon x started 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.? 172.21.15.156:0/1030161332' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T16:45:33.320 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:33 smithi156 ceph-mon[65097]: mgrmap e47: y(active, since 1.00998s), standbys: x 2024-04-24T16:45:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:34 smithi156 ceph-mon[65097]: [24/Apr/2024:16:45:32] ENGINE Bus STARTING 2024-04-24T16:45:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:34 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:34 smithi156 ceph-mon[65097]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:34 smithi156 ceph-mon[65097]: [24/Apr/2024:16:45:33] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:45:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:34 smithi156 ceph-mon[65097]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state']) 2024-04-24T16:45:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:34 smithi156 ceph-mon[65097]: Cluster is now healthy 2024-04-24T16:45:34.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[26293]: [24/Apr/2024:16:45:32] ENGINE Bus STARTING 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[26293]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[26293]: [24/Apr/2024:16:45:33] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[26293]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state']) 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[26293]: Cluster is now healthy 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[81602]: [24/Apr/2024:16:45:32] ENGINE Bus STARTING 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[81602]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[81602]: [24/Apr/2024:16:45:33] ENGINE Serving on https://172.21.15.70:7150 2024-04-24T16:45:34.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[81602]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state']) 2024-04-24T16:45:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:34 smithi070 ceph-mon[81602]: Cluster is now healthy 2024-04-24T16:45:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[81602]: [24/Apr/2024:16:45:33] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:45:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[81602]: [24/Apr/2024:16:45:33] ENGINE Bus STARTED 2024-04-24T16:45:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[81602]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[81602]: mgrmap e48: y(active, since 2s), standbys: x 2024-04-24T16:45:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[26293]: [24/Apr/2024:16:45:33] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:45:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[26293]: [24/Apr/2024:16:45:33] ENGINE Bus STARTED 2024-04-24T16:45:35.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[26293]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:35.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:35 smithi070 ceph-mon[26293]: mgrmap e48: y(active, since 2s), standbys: x 2024-04-24T16:45:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:35 smithi156 ceph-mon[65097]: [24/Apr/2024:16:45:33] ENGINE Serving on http://172.21.15.70:8765 2024-04-24T16:45:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:35 smithi156 ceph-mon[65097]: [24/Apr/2024:16:45:33] ENGINE Bus STARTED 2024-04-24T16:45:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:35 smithi156 ceph-mon[65097]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:35 smithi156 ceph-mon[65097]: mgrmap e48: y(active, since 2s), standbys: x 2024-04-24T16:45:36.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:45:36] "GET /metrics HTTP/1.1" 200 35211 "" "Prometheus/2.43.0" 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[26293]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[26293]: mgrmap e49: y(active, since 4s), standbys: x 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[81602]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[81602]: mgrmap e49: y(active, since 4s), standbys: x 2024-04-24T16:45:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:37 smithi156 ceph-mon[65097]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:37 smithi156 ceph-mon[65097]: mgrmap e49: y(active, since 4s), standbys: x 2024-04-24T16:45:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:38.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:38.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:38.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:38.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:38.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:38 smithi156 ceph-mon[65097]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[26293]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:39.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:39.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:38 smithi070 ceph-mon[81602]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:45:41.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:41 smithi070 ceph-mon[26293]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:45:41.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:41 smithi070 ceph-mon[81602]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:45:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:41 smithi156 ceph-mon[65097]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-24T16:45:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:42 smithi156 ceph-mon[65097]: Detected new or changed devices on smithi070 2024-04-24T16:45:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:42.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[26293]: Detected new or changed devices on smithi070 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[81602]: Detected new or changed devices on smithi070 2024-04-24T16:45:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:45:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:45:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:45:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:45:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:45:43.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:45:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-24T16:45:43.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi156:/etc/ceph/ceph.conf 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Updating smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:43.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:45:43.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:43.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:43.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-24T16:45:43.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-24T16:45:44.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[26293]: Upgrade: It appears safe to stop mon.c 2024-04-24T16:45:44.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[26293]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:44.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:44.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:44.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:44.349 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[26293]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:44.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[81602]: Upgrade: It appears safe to stop mon.c 2024-04-24T16:45:44.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:44.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:44.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:44.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:44.350 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:44 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:44 smithi156 ceph-mon[65097]: Upgrade: It appears safe to stop mon.c 2024-04-24T16:45:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:44 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:44 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:44 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:44 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:44 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:45.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-mon[26293]: Upgrade: Updating mon.c 2024-04-24T16:45:45.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-mon[26293]: Deploying daemon mon.c on smithi070 2024-04-24T16:45:45.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-mon[26293]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-24T16:45:45.214 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-mon[81602]: Upgrade: Updating mon.c 2024-04-24T16:45:45.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-mon[81602]: Deploying daemon mon.c on smithi070 2024-04-24T16:45:45.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-mon[81602]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-24T16:45:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:45 smithi156 ceph-mon[65097]: Upgrade: Updating mon.c 2024-04-24T16:45:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:45 smithi156 ceph-mon[65097]: Deploying daemon mon.c on smithi070 2024-04-24T16:45:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:45 smithi156 ceph-mon[65097]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-24T16:45:45.736 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:45 smithi070 systemd[1]: Stopping Ceph mon.c for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:45:46.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c[26289]: 2024-04-24T16:45:45.734+0000 7fa0f876f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:45:46.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c[26289]: 2024-04-24T16:45:45.734+0000 7fa0f876f700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-24T16:45:46.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:45 smithi070 podman[86331]: 2024-04-24 16:45:45.898518403 +0000 UTC m=+0.256800631 container died 2b11cd568406b08c479d0dac41dda90a05602e5bc3e9b024e45cb0cba1c33d96 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, version=8, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, name=centos-stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-24T16:45:46.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:46 smithi070 podman[86331]: 2024-04-24 16:45:46.302508373 +0000 UTC m=+0.660790593 container cleanup 2b11cd568406b08c479d0dac41dda90a05602e5bc3e9b024e45cb0cba1c33d96 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, release=754, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8) 2024-04-24T16:45:46.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:46 smithi070 bash[86331]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c 2024-04-24T16:45:46.777 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:45:46] "GET /metrics HTTP/1.1" 200 35211 "" "Prometheus/2.43.0" 2024-04-24T16:45:46.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:46 smithi070 podman[86346]: 2024-04-24 16:45:46.52909052 +0000 UTC m=+0.628150576 container remove 2b11cd568406b08c479d0dac41dda90a05602e5bc3e9b024e45cb0cba1c33d96 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, name=centos-stream, io.openshift.expose-services=, release=754, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.buildah.version=1.19.8, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-24T16:45:46.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:46 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-24T16:45:46.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:46 smithi070 systemd[1]: Stopped Ceph mon.c for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:45:46.777 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:46 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.c.service: Consumed 17.076s CPU time. 2024-04-24T16:45:47.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:46 smithi070 systemd[1]: Starting Ceph mon.c for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:45:47.422 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 podman[86477]: 2024-04-24 16:45:47.075857922 +0000 UTC m=+0.018124406 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:45:47.422 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 podman[86477]: 2024-04-24 16:45:47.180693399 +0000 UTC m=+0.122959914 container create ea92875c6eecf9b15740d388637bb66b82e9dcc2aec03b0f53f46e29a8581ead (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-24T16:45:47.422 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 podman[86477]: 2024-04-24 16:45:47.392779816 +0000 UTC m=+0.335046290 container init ea92875c6eecf9b15740d388637bb66b82e9dcc2aec03b0f53f46e29a8581ead (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T16:45:47.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 podman[86477]: 2024-04-24 16:45:47.396125614 +0000 UTC m=+0.338392090 container start ea92875c6eecf9b15740d388637bb66b82e9dcc2aec03b0f53f46e29a8581ead (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-24T16:45:47.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T16:45:47.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev), process ceph-mon, pid 2 2024-04-24T16:45:47.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: pidfile_write: ignore empty --pid-file 2024-04-24T16:45:47.423 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: load: jerasure load: lrc 2024-04-24T16:45:47.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: RocksDB version: 7.9.2 2024-04-24T16:45:47.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Git sha 0 2024-04-24T16:45:47.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Compile date 2024-04-24 13:56:12 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: DB SUMMARY 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: DB Session ID: R1U97YBNTCU3O0JDU5YJ 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: CURRENT file: CURRENT 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3232 Bytes 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2455157 ; 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.error_if_exists: 0 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.create_if_missing: 0 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.paranoid_checks: 1 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T16:45:47.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.env: 0x560d5cd21cc0 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.info_log: 0x560d5e43bd80 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.statistics: (nil) 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.use_fsync: 0 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_log_file_size: 0 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.allow_fallocate: 1 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T16:45:47.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.use_direct_reads: 0 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.db_log_dir: 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.wal_dir: 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T16:45:47.796 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.write_buffer_manager: 0x560d5e40d220 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.unordered_write: 0 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T16:45:47.797 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.row_cache: None 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.wal_filter: None 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.two_write_queues: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.wal_compression: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.atomic_flush: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.log_readahead_size: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T16:45:47.798 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T16:45:47.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T16:45:47.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T16:45:47.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T16:45:47.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_background_jobs: 2 2024-04-24T16:45:47.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_background_compactions: -1 2024-04-24T16:45:47.799 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_subcompactions: 1 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T16:45:47.800 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_open_files: -1 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_background_flushes: -1 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Compression algorithms supported: 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kZSTD supported: 0 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kXpressCompression supported: 0 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kZlibCompression supported: 1 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kSnappyCompression supported: 1 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kLZ4Compression supported: 1 2024-04-24T16:45:47.801 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: kBZip2Compression supported: 0 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.merge_operator: 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_filter: None 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d5e43bf40) 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-04-24T16:45:47.802 INFO:journalctl@ceph.mon.c.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_type: 0 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_index_type: 0 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_shortening: 1 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: checksum: 4 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: no_block_cache: 0 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache: 0x560d5e43f1f0 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_options: 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: capacity : 536870912 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: num_shard_bits : 4 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: strict_capacity_limit : 0 2024-04-24T16:45:47.803 INFO:journalctl@ceph.mon.c.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_cache_compressed: (nil) 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: persistent_cache: (nil) 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size: 4096 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_size_deviation: 10 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_restart_interval: 16 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: index_block_restart_interval: 1 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: metadata_block_size: 4096 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: partition_filters: 0 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: use_delta_encoding: 1 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: filter_policy: bloomfilter 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: whole_key_filtering: 1 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: verify_compression: 0 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: format_version: 5 2024-04-24T16:45:47.804 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_index_compression: 1 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout: block_align: 0 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout: max_auto_readahead_size: 262144 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout: prepopulate_block_cache: 0 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout: initial_auto_readahead_size: 8192 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression: NoCompression 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.num_levels: 7 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T16:45:47.805 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T16:45:47.806 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T16:45:47.807 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T16:45:47.808 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.inplace_update_support: 0 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T16:45:47.809 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.bloom_locality: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.max_successive_merges: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.ttl: 2592000 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.enable_blob_files: false 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.min_blob_size: 0 2024-04-24T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32427, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T16:45:47.811 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2a829568-bc51-4ade-8565-e10bd2a09140 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977147425700, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977147431976, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1550027, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32432, "largest_seqno": 32853, "table_properties": {"data_size": 1547031, "index_size": 1444, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4875, "raw_average_key_size": 24, "raw_value_size": 1542410, "raw_average_value_size": 7635, "num_data_blocks": 65, "num_entries": 202, "num_filter_entries": 202, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713977147, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2a829568-bc51-4ade-8565-e10bd2a09140", "db_session_id": "R1U97YBNTCU3O0JDU5YJ", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713977147432080, "job": 1, "event": "recovery_finished"} 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560d5e55c000 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: DB pointer 0x560d5e548000 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** DB Stats ** 2024-04-24T16:45:47.812 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: L0 1/0 1.48 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 505.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: L6 1/0 10.62 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Sum 2/0 12.09 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 505.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:45:47.813 INFO:journalctl@ceph.mon.c.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 505.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 505.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T16:45:47.814 INFO:journalctl@ceph.mon.c.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: Cumulative compaction: 0.00 GB write, 172.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: Interval compaction: 0.00 GB write, 172.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: Block cache BinnedLRUCache@0x560d5e43f1f0#2 capacity: 512.00 MB usage: 2.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.58 KB,0.000301003%) Misc(1,0.00 KB,0%) 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: starting mon.c rank 1 at public addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] at bind addrs [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???) e3 preinit fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???).mds e1 new map 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???).mds e1 print_map 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: e1 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T16:45:47.815 INFO:journalctl@ceph.mon.c.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: legacy client fscid: -1 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout: No filesystems configured 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 ceph-mon[86496]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 bash[86477]: ea92875c6eecf9b15740d388637bb66b82e9dcc2aec03b0f53f46e29a8581ead 2024-04-24T16:45:47.816 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:47 smithi070 systemd[1]: Started Ceph mon.c for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:45:48.135 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:45:48.783 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:45:48.783 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (2m) 12s ago 24m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:45:48.783 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (2m) 12s ago 24m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:45:48.783 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 12s ago 23m 58.8M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:45:48.783 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (3m) 12s ago 28m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (13m) 12s ago 30m 474M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (27s) 12s ago 30m 30.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (77s) 12s ago 29m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 starting - - - 2048M 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (3m) 12s ago 25m 16.4M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (3m) 12s ago 25m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (28m) 12s ago 28m 57.2M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (28m) 12s ago 28m 58.3M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (27m) 12s ago 27m 50.9M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (27m) 12s ago 27m 52.5M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:45:48.784 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (27m) 12s ago 27m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:45:48.785 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (26m) 12s ago 26m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:45:48.785 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (26m) 12s ago 26m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:45:48.785 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (25m) 12s ago 25m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:45:48.785 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (2m) 12s ago 25m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:45:48.785 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (23m) 12s ago 23m 76.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:45:48.785 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (23m) 12s ago 23m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:49.195 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:45:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:45:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:45:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:45:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-24T16:45:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 5 2024-04-24T16:45:49.196 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:45:49.196 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:45:49.549 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:45:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:45:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:45:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi070", 2024-04-24T16:45:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:45:49.549 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "1/2 daemons upgraded", 2024-04-24T16:45:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-04-24T16:45:49.550 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:45:49.550 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:45:50.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='client.54135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='client.54141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='client.54147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: fsmap 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: mgrmap e49: y(active, since 17s), standbys: x 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: mon.c calling monitor election 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: mon.b calling monitor election 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: mon.a calling monitor election 2024-04-24T16:45:50.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: monmap epoch 4 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: last_changed 2024-04-24T16:45:49.046241+0000 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: created 2024-04-24T16:15:12.900488+0000 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: min_mon_release 19 (squid) 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: election_strategy: 1 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: 1: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: fsmap 2024-04-24T16:45:50.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: mgrmap e49: y(active, since 17s), standbys: x 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1934907927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='client.54135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='client.54141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='client.54147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: fsmap 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: mgrmap e49: y(active, since 17s), standbys: x 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: mon.c calling monitor election 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: mon.b calling monitor election 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: mon.a calling monitor election 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: monmap epoch 4 2024-04-24T16:45:50.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: last_changed 2024-04-24T16:45:49.046241+0000 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: created 2024-04-24T16:15:12.900488+0000 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: min_mon_release 19 (squid) 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: election_strategy: 1 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: 1: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: fsmap 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: mgrmap e49: y(active, since 17s), standbys: x 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:50.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:50.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:50.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:50 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1934907927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:50.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='client.54135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='client.54141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='client.54147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:50.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:50.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: fsmap 2024-04-24T16:45:50.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:50.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: mgrmap e49: y(active, since 17s), standbys: x 2024-04-24T16:45:50.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:45:50.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: mon.c calling monitor election 2024-04-24T16:45:50.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: mon.b calling monitor election 2024-04-24T16:45:50.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: mon.a calling monitor election 2024-04-24T16:45:50.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T16:45:50.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T16:45:50.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T16:45:50.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T16:45:50.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: monmap epoch 4 2024-04-24T16:45:50.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: fsid be31be9a-0255-11ef-bc93-c7b262605968 2024-04-24T16:45:50.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: last_changed 2024-04-24T16:45:49.046241+0000 2024-04-24T16:45:50.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: created 2024-04-24T16:15:12.900488+0000 2024-04-24T16:45:50.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: min_mon_release 19 (squid) 2024-04-24T16:45:50.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: election_strategy: 1 2024-04-24T16:45:50.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-24T16:45:50.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: 1: [v2:172.21.15.70:3301/0,v1:172.21.15.70:6790/0] mon.c 2024-04-24T16:45:50.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T16:45:50.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: fsmap 2024-04-24T16:45:50.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: osdmap e112: 8 total, 8 up, 8 in 2024-04-24T16:45:50.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: mgrmap e49: y(active, since 17s), standbys: x 2024-04-24T16:45:50.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:45:50.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:45:50.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:50.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:50.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:45:50.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:50 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1934907927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:45:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:51 smithi156 ceph-mon[65097]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:51 smithi156 ceph-mon[65097]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-04-24T16:45:51.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:51 smithi070 ceph-mon[86496]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:51.532 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:51 smithi070 ceph-mon[86496]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-04-24T16:45:51.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:51 smithi070 ceph-mon[81602]: from='client.54159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:45:51.532 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:51 smithi070 ceph-mon[81602]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-04-24T16:45:52.440 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:52 smithi070 ceph-mon[86496]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-04-24T16:45:52.441 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:52 smithi070 ceph-mon[81602]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-04-24T16:45:52.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:52 smithi156 ceph-mon[65097]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-04-24T16:45:52.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:52 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:45:52.439+0000 7f22938a6700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T16:45:53.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:53 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:53.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:53 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:53.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:53 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:45:54.792 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:54 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:54.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:54 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:54.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:54 smithi070 ceph-mon[86496]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:54.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:54 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:54.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:54 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:54.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:54 smithi070 ceph-mon[81602]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:54.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:54 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:54.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:54 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:54.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:54 smithi156 ceph-mon[65097]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:45:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:55 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:55.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:55 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:55.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:55 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:55.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:55 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:55.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:55 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:55.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:55 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:56.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:56 smithi070 ceph-mon[81602]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:45:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:45:56] "GET /metrics HTTP/1.1" 200 38015 "" "Prometheus/2.43.0" 2024-04-24T16:45:56.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:56 smithi070 ceph-mon[86496]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:56.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:56 smithi156 ceph-mon[65097]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:45:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: Reconfiguring mon.a (monmap changed)... 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:59.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:45:58 smithi156 ceph-mon[65097]: Reconfiguring daemon mon.a on smithi070 2024-04-24T16:45:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-04-24T16:45:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-04-24T16:45:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: Reconfiguring mon.a (monmap changed)... 2024-04-24T16:45:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:59.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:59.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[81602]: Reconfiguring daemon mon.a on smithi070 2024-04-24T16:45:59.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:59.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:45:59.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-04-24T16:45:59.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 523 B/s rd, 0 op/s 2024-04-24T16:45:59.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:45:59.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: Reconfiguring mon.a (monmap changed)... 2024-04-24T16:45:59.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:45:59.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:45:59.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:45:59.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:45:58 smithi070 ceph-mon[86496]: Reconfiguring daemon mon.a on smithi070 2024-04-24T16:46:00.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:00.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:00.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: Reconfiguring mgr.y (monmap changed)... 2024-04-24T16:46:00.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:46:00.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:46:00.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:00.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: Reconfiguring daemon mgr.y on smithi070 2024-04-24T16:46:00.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:00 smithi156 ceph-mon[65097]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:00.996 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: Reconfiguring mgr.y (monmap changed)... 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: Reconfiguring daemon mgr.y on smithi070 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[86496]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: Reconfiguring mgr.y (monmap changed)... 2024-04-24T16:46:00.997 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:46:00.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:46:00.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:00.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: Reconfiguring daemon mgr.y on smithi070 2024-04-24T16:46:00.998 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:00 smithi070 ceph-mon[81602]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: Reconfiguring mon.c (monmap changed)... 2024-04-24T16:46:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:46:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:46:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: Reconfiguring daemon mon.c on smithi070 2024-04-24T16:46:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:02.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:02.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:02.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:02.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: Reconfiguring mon.c (monmap changed)... 2024-04-24T16:46:02.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:46:02.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:46:02.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: Reconfiguring daemon mon.c on smithi070 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: Reconfiguring mon.c (monmap changed)... 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:02.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: Reconfiguring daemon mon.c on smithi070 2024-04-24T16:46:02.795 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:02.795 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[81602]: Reconfiguring osd.0 (monmap changed)... 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.0 on smithi070 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:04.287 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:04.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:04.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[86496]: Reconfiguring osd.0 (monmap changed)... 2024-04-24T16:46:04.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:46:04.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:04.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.0 on smithi070 2024-04-24T16:46:04.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:03 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:03 smithi156 ceph-mon[65097]: Reconfiguring osd.0 (monmap changed)... 2024-04-24T16:46:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:46:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:03 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.0 on smithi070 2024-04-24T16:46:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:03 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[86496]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:46:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:05.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[81602]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:05.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:05.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:05.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:46:05.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:04 smithi156 ceph-mon[65097]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:46:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:06.243 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:05 smithi070 ceph-mon[81602]: Reconfiguring osd.1 (monmap changed)... 2024-04-24T16:46:06.243 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:05 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.1 on smithi070 2024-04-24T16:46:06.243 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:05 smithi070 ceph-mon[86496]: Reconfiguring osd.1 (monmap changed)... 2024-04-24T16:46:06.244 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:05 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.1 on smithi070 2024-04-24T16:46:06.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:05 smithi156 ceph-mon[65097]: Reconfiguring osd.1 (monmap changed)... 2024-04-24T16:46:06.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:05 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.1 on smithi070 2024-04-24T16:46:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:46:06] "GET /metrics HTTP/1.1" 200 38021 "" "Prometheus/2.43.0" 2024-04-24T16:46:07.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[86496]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:07.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:07.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:07.264 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:46:07.265 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:07.265 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[81602]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:07.265 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:07.265 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:07.265 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:46:07.265 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:06 smithi156 ceph-mon[65097]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:46:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[81602]: Reconfiguring osd.2 (monmap changed)... 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.2 on smithi070 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[86496]: Reconfiguring osd.2 (monmap changed)... 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.2 on smithi070 2024-04-24T16:46:08.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:08.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:08.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:46:08.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:07 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:07 smithi156 ceph-mon[65097]: Reconfiguring osd.2 (monmap changed)... 2024-04-24T16:46:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:07 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.2 on smithi070 2024-04-24T16:46:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:07 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:08.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:07 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:08.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:07 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:46:08.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:07 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:09.292 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:08 smithi070 ceph-mon[81602]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:46:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:08 smithi070 ceph-mon[81602]: Reconfiguring osd.3 (monmap changed)... 2024-04-24T16:46:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:08 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.3 on smithi070 2024-04-24T16:46:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:08 smithi070 ceph-mon[86496]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:46:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:08 smithi070 ceph-mon[86496]: Reconfiguring osd.3 (monmap changed)... 2024-04-24T16:46:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:08 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.3 on smithi070 2024-04-24T16:46:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:08 smithi156 ceph-mon[65097]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:46:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:08 smithi156 ceph-mon[65097]: Reconfiguring osd.3 (monmap changed)... 2024-04-24T16:46:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:08 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.3 on smithi070 2024-04-24T16:46:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[86496]: Reconfiguring rgw.foo.smithi070.vhdldq (monmap changed)... 2024-04-24T16:46:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:46:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[86496]: Reconfiguring daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:46:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[86496]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:10.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:10.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:10.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[81602]: Reconfiguring rgw.foo.smithi070.vhdldq (monmap changed)... 2024-04-24T16:46:10.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:46:10.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:10.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[81602]: Reconfiguring daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:46:10.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:10 smithi070 ceph-mon[81602]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:10.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:10.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:10.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:10 smithi156 ceph-mon[65097]: Reconfiguring rgw.foo.smithi070.vhdldq (monmap changed)... 2024-04-24T16:46:10.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:46:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:10 smithi156 ceph-mon[65097]: Reconfiguring daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:46:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:10 smithi156 ceph-mon[65097]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: Reconfiguring mon.b (monmap changed)... 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: Reconfiguring daemon mon.b on smithi156 2024-04-24T16:46:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:12 smithi156 ceph-mon[65097]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: Reconfiguring mon.b (monmap changed)... 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: Reconfiguring daemon mon.b on smithi156 2024-04-24T16:46:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[86496]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: Reconfiguring mon.b (monmap changed)... 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: Reconfiguring daemon mon.b on smithi156 2024-04-24T16:46:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:12 smithi070 ceph-mon[81602]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: Reconfiguring mgr.x (monmap changed)... 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: Reconfiguring daemon mgr.x on smithi156 2024-04-24T16:46:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: Reconfiguring mgr.x (monmap changed)... 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: Reconfiguring daemon mgr.x on smithi156 2024-04-24T16:46:14.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:13 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: Reconfiguring mgr.x (monmap changed)... 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: Reconfiguring daemon mgr.x on smithi156 2024-04-24T16:46:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:13 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[81602]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[86496]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:46:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:15.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:14 smithi156 ceph-mon[65097]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:15.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:15.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:15.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:46:15.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:15 smithi156 ceph-mon[65097]: Reconfiguring osd.4 (monmap changed)... 2024-04-24T16:46:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:15 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.4 on smithi156 2024-04-24T16:46:16.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:15 smithi070 ceph-mon[86496]: Reconfiguring osd.4 (monmap changed)... 2024-04-24T16:46:16.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:15 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.4 on smithi156 2024-04-24T16:46:16.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:15 smithi070 ceph-mon[81602]: Reconfiguring osd.4 (monmap changed)... 2024-04-24T16:46:16.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:15 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.4 on smithi156 2024-04-24T16:46:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:46:16] "GET /metrics HTTP/1.1" 200 38021 "" "Prometheus/2.43.0" 2024-04-24T16:46:17.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:16 smithi156 ceph-mon[65097]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:17.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:17.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:17.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:16 smithi156 ceph-mon[65097]: Reconfiguring osd.5 (monmap changed)... 2024-04-24T16:46:17.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:46:17.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:17.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:16 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.5 on smithi156 2024-04-24T16:46:17.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[81602]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:17.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:17.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[81602]: Reconfiguring osd.5 (monmap changed)... 2024-04-24T16:46:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:46:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.5 on smithi156 2024-04-24T16:46:17.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[86496]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:17.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:17.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:17.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[86496]: Reconfiguring osd.5 (monmap changed)... 2024-04-24T16:46:17.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:46:17.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:17.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:16 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.5 on smithi156 2024-04-24T16:46:18.237 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:18.237 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:18.237 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:18.237 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:46:18.237 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:46:18.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:19.291 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:18 smithi156 ceph-mon[65097]: Reconfiguring osd.6 (monmap changed)... 2024-04-24T16:46:19.291 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:18 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.6 on smithi156 2024-04-24T16:46:19.291 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:18 smithi156 ceph-mon[65097]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:18 smithi070 ceph-mon[86496]: Reconfiguring osd.6 (monmap changed)... 2024-04-24T16:46:19.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:18 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.6 on smithi156 2024-04-24T16:46:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:18 smithi070 ceph-mon[86496]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:18 smithi070 ceph-mon[81602]: Reconfiguring osd.6 (monmap changed)... 2024-04-24T16:46:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:18 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.6 on smithi156 2024-04-24T16:46:19.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:18 smithi070 ceph-mon[81602]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:19.904 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (2m) 27s ago 25m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (2m) 43s ago 24m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 27s ago 24m 61.5M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (4m) 43s ago 29m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (14m) 27s ago 30m 496M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (58s) 27s ago 31m 39.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (109s) 43s ago 29m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (33s) 27s ago 29m 22.6M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (3m) 27s ago 25m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (3m) 43s ago 25m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:46:20.561 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (29m) 27s ago 29m 56.2M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (28m) 27s ago 28m 58.3M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (28m) 27s ago 28m 51.0M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (27m) 27s ago 27m 52.7M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (27m) 43s ago 27m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (27m) 43s ago 27m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (26m) 43s ago 26m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (26m) 43s ago 26m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (3m) 43s ago 25m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (24m) 27s ago 24m 77.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:46:20.562 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (24m) 43s ago 24m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: Reconfiguring osd.7 (monmap changed)... 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: Reconfiguring daemon osd.7 on smithi156 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:20 smithi156 ceph-mon[65097]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: Reconfiguring osd.7 (monmap changed)... 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: Reconfiguring daemon osd.7 on smithi156 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[86496]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:20.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:20.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:20.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: Reconfiguring osd.7 (monmap changed)... 2024-04-24T16:46:20.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:46:20.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:20.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: Reconfiguring daemon osd.7 on smithi156 2024-04-24T16:46:20.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:20.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:20 smithi070 ceph-mon[81602]: from='client.54165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:20.962 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:46:20.963 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-24T16:46:20.964 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 5 2024-04-24T16:46:20.964 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:46:20.964 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:46:21.297 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:46:21.297 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:46:21.297 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi070", 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: "mon" 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "2/2 daemons upgraded", 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading mon daemons", 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:46:21.298 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:46:22.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:22.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: from='client.54177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:22.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:22.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:22.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: Reconfiguring rgw.foo.smithi156.pmtzwp (monmap changed)... 2024-04-24T16:46:22.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:46:22.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:22.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: Reconfiguring daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:46:22.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:21 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3967503880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: from='client.54177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: Reconfiguring rgw.foo.smithi156.pmtzwp (monmap changed)... 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: Reconfiguring daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3967503880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: from='client.54177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: Reconfiguring rgw.foo.smithi156.pmtzwp (monmap changed)... 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:22.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: Reconfiguring daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:46:22.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:21 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3967503880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-24T16:46:23.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:46:23.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:46:23.165 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:23.166 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:46:23.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:23.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:46:23.167 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:23.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.298 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:46:23.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:23.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:46:23.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:46:23.304 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:46:23.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:23.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:46:23.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:46:23.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:46:23.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mon 2024-04-24T16:46:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all crash 2024-04-24T16:46:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mds 2024-04-24T16:46:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:46:24.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nfs 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all node-exporter 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all prometheus 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all alertmanager 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all grafana 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all loki 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all promtail 2024-04-24T16:46:24.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Finalizing container_image settings 2024-04-24T16:46:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: Upgrade: Complete! 2024-04-24T16:46:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-04-24T16:46:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-04-24T16:46:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2024-04-24T16:46:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:24.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:23 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mon 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all crash 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mds 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nfs 2024-04-24T16:46:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all node-exporter 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all prometheus 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all alertmanager 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all grafana 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all loki 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all promtail 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Finalizing container_image settings 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: Upgrade: Complete! 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:24.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mon 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all crash 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mds 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nfs 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all node-exporter 2024-04-24T16:46:24.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all prometheus 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all alertmanager 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all grafana 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all loki 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all promtail 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Finalizing container_image settings 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: Upgrade: Complete! 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 735 B/s rd, 0 op/s 2024-04-24T16:46:24.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:24.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:23 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:26.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:25 smithi070 ceph-mon[81602]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:26.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:25 smithi070 ceph-mon[86496]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:26.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:25 smithi156 ceph-mon[65097]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:26.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:46:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:46:26] "GET /metrics HTTP/1.1" 200 38016 "" "Prometheus/2.43.0" 2024-04-24T16:46:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:28 smithi156 ceph-mon[65097]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:28 smithi070 ceph-mon[86496]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:28.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:28.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:28 smithi070 ceph-mon[81602]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:46:28.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:46:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:30 smithi156 ceph-mon[65097]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:46:30.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:30 smithi070 ceph-mon[86496]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:46:30.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:30 smithi070 ceph-mon[81602]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:46:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:32 smithi156 ceph-mon[65097]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:32.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:32 smithi070 ceph-mon[86496]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:32.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:32.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:32 smithi070 ceph-mon[81602]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:32.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:34 smithi156 ceph-mon[65097]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:46:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:34 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:34 smithi070 ceph-mon[86496]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:46:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:34 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:34.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:34 smithi070 ceph-mon[81602]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:46:34.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:34 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:36.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:36 smithi156 ceph-mon[65097]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:36.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:36 smithi070 ceph-mon[86496]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:36.519 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:36 smithi070 ceph-mon[81602]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:46:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:46:36] "GET /metrics HTTP/1.1" 200 38021 "" "Prometheus/2.43.0" 2024-04-24T16:46:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:38 smithi156 ceph-mon[65097]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:38.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:38 smithi070 ceph-mon[81602]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:38.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:38 smithi070 ceph-mon[86496]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:40.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:40 smithi156 ceph-mon[65097]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:40 smithi070 ceph-mon[81602]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:40 smithi070 ceph-mon[86496]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:42 smithi156 ceph-mon[65097]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:42.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:42 smithi070 ceph-mon[86496]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:42 smithi070 ceph-mon[81602]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:44 smithi156 ceph-mon[65097]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:44 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:44.444 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:44 smithi070 ceph-mon[86496]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:44.444 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:44 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:44.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:44 smithi070 ceph-mon[81602]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:44.444 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:44 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:46.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:46 smithi070 ceph-mon[81602]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:46.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:46 smithi070 ceph-mon[86496]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:46 smithi156 ceph-mon[65097]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:46:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:46:46] "GET /metrics HTTP/1.1" 200 38021 "" "Prometheus/2.43.0" 2024-04-24T16:46:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:47.542 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:46:48.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:48 smithi156 ceph-mon[65097]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:48.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:48 smithi070 ceph-mon[86496]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:48.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:48 smithi070 ceph-mon[81602]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:50 smithi156 ceph-mon[65097]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:50.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:50 smithi070 ceph-mon[86496]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:50.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:50 smithi070 ceph-mon[81602]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:52.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:52 smithi070 ceph-mon[86496]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:52.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:52 smithi070 ceph-mon[81602]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:52 smithi156 ceph-mon[65097]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:52.430 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:46:53.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:53 smithi070 ceph-mon[86496]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:53.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:53 smithi070 ceph-mon[81602]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:53 smithi156 ceph-mon[65097]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:54 smithi070 ceph-mon[81602]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:54 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:54 smithi070 ceph-mon[86496]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:54 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:54.324 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:46:54.324 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 60s ago 25m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:46:54.324 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (3m) 77s ago 25m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:46:54.324 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 60s ago 24m 61.5M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:46:54.324 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (4m) 77s ago 29m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (14m) 60s ago 31m 496M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (92s) 60s ago 31m 39.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (2m) 77s ago 30m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (67s) 60s ago 30m 22.6M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (4m) 60s ago 26m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (4m) 77s ago 26m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (29m) 60s ago 29m 56.2M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (29m) 60s ago 29m 58.3M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (28m) 60s ago 28m 51.0M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (28m) 60s ago 28m 52.7M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (28m) 77s ago 28m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (27m) 77s ago 27m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:46:54.325 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (27m) 77s ago 27m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:46:54.326 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (26m) 77s ago 26m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:46:54.326 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (3m) 77s ago 26m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:46:54.326 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (24m) 60s ago 24m 77.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:46:54.326 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (24m) 77s ago 24m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:46:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:54 smithi156 ceph-mon[65097]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:54 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:46:54.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-04-24T16:46:56.226 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:56 smithi070 ceph-mon[86496]: from='client.54201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:56.227 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:56 smithi070 ceph-mon[86496]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:56.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:56 smithi070 ceph-mon[81602]: from='client.54201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:56.227 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:56 smithi070 ceph-mon[81602]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:56 smithi156 ceph-mon[65097]: from='client.54201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:46:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:56 smithi156 ceph-mon[65097]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:46:56.792 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:46:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:46:56] "GET /metrics HTTP/1.1" 200 38018 "" "Prometheus/2.43.0" 2024-04-24T16:46:56.861 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:46:57.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:57 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2182466276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:57 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2182466276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:57 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2182466276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:46:57.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-04-24T16:46:58.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:46:58 smithi156 ceph-mon[65097]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:58.508 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:46:58 smithi070 ceph-mon[86496]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:58.508 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:46:58 smithi070 ceph-mon[81602]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:46:59.505 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)" 2024-04-24T16:47:00.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-04-24T16:47:00.363 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:00 smithi070 ceph-mon[86496]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:00.363 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:00 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1566086365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:00.363 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:00 smithi070 ceph-mon[81602]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:00.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:00 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1566086365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:00 smithi156 ceph-mon[65097]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:00 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1566086365' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:02 smithi156 ceph-mon[65097]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:02.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:02 smithi070 ceph-mon[86496]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:02.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:02.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:02 smithi070 ceph-mon[81602]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:02.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:03.364 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:03 smithi070 ceph-mon[86496]: from='client.54219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:03.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:03 smithi070 ceph-mon[81602]: from='client.54219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:03 smithi156 ceph-mon[65097]: from='client.54219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:04.402 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:04 smithi070 ceph-mon[81602]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:04.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:04 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:04.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:04 smithi070 ceph-mon[86496]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:04.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:04 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:04 smithi156 ceph-mon[65097]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:04 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:04.470 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:47:05.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:47:06.372 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:06 smithi070 ceph-mon[86496]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:06.373 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:06 smithi070 ceph-mon[81602]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:06.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:06 smithi156 ceph-mon[65097]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:06.659 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:47:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:47:06] "GET /metrics HTTP/1.1" 200 38019 "" "Prometheus/2.43.0" 2024-04-24T16:47:07.109 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": null, 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": false, 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "", 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": null, 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:47:07.110 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:47:07.837 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:47:08.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:08 smithi070 ceph-mon[86496]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:08.370 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:08 smithi070 ceph-mon[86496]: from='client.54225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:08.371 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:08 smithi070 ceph-mon[81602]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:08.371 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:08 smithi070 ceph-mon[81602]: from='client.54225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:08 smithi156 ceph-mon[65097]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:08 smithi156 ceph-mon[65097]: from='client.54225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:09.796 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:47:09.796 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:47:09.796 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:47:10.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:10 smithi156 ceph-mon[65097]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:10.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:10 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3096415459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:47:10.438 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:10 smithi070 ceph-mon[81602]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:10.438 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:10 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3096415459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:47:10.438 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:10 smithi070 ceph-mon[86496]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:10.438 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:10 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3096415459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:47:10.461 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-04-24T16:47:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:12 smithi156 ceph-mon[65097]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:12.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:12 smithi070 ceph-mon[81602]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:12 smithi070 ceph-mon[86496]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:14.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:14 smithi070 ceph-mon[86496]: from='client.54237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:14.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:14 smithi070 ceph-mon[86496]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:14.390 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:14 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:14.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:14 smithi070 ceph-mon[81602]: from='client.54237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:14.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:14 smithi070 ceph-mon[81602]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:14.391 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:14 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:14 smithi156 ceph-mon[65097]: from='client.54237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:14 smithi156 ceph-mon[65097]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:14 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:14.444 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:47:15.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-24T16:47:15.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:47:15.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:15.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:15.589 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.590 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.591 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:47:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:15.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:16.640 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:47:16] "GET /metrics HTTP/1.1" 200 38019 "" "Prometheus/2.43.0" 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[86496]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[86496]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:47:16.641 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[81602]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-04-24T16:47:16.642 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[81602]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:16.642 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:47:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:16 smithi156 ceph-mon[65097]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-04-24T16:47:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:16 smithi156 ceph-mon[65097]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:17.146 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:47:17.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:17 smithi070 ceph-mon[86496]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:47:17.719 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:17.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:17 smithi070 ceph-mon[81602]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:47:17.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:17.822 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:47:17.822 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (3m) 84s ago 26m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:47:17.822 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (3m) 101s ago 25m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:47:17.822 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 84s ago 25m 61.5M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (5m) 101s ago 30m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (15m) 84s ago 31m 496M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (116s) 84s ago 32m 39.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (2m) 101s ago 30m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (90s) 84s ago 30m 22.6M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (4m) 84s ago 26m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (4m) 101s ago 26m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (30m) 84s ago 30m 56.2M 2992M 17.2.0 e1d6a67b021e 5c223a67a387 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (29m) 84s ago 29m 58.3M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (29m) 84s ago 29m 51.0M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (28m) 84s ago 28m 52.7M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (28m) 101s ago 28m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (28m) 101s ago 28m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:47:17.823 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (27m) 101s ago 27m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:47:17.824 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (27m) 101s ago 27m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:47:17.824 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (4m) 101s ago 26m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:47:17.824 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (25m) 84s ago 25m 77.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:47:17.824 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (25m) 101s ago 25m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:47:17.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:17 smithi156 ceph-mon[65097]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:47:17.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:18.232 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:47:18.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:18.234 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:47:18.234 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-24T16:47:18.234 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 5 2024-04-24T16:47:18.234 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:47:18.234 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[81602]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[81602]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[81602]: from='client.54255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[81602]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3621145385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[86496]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[86496]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[86496]: from='client.54255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[86496]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:18.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:18 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3621145385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:18.572 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:47:18.572 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", 2024-04-24T16:47:18.572 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:47:18.573 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-04-24T16:47:18.573 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:47:18.573 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "", 2024-04-24T16:47:18.573 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc image", 2024-04-24T16:47:18.573 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:47:18.573 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:47:18.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:18 smithi156 ceph-mon[65097]: from='client.54243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:18 smithi156 ceph-mon[65097]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:18 smithi156 ceph-mon[65097]: from='client.54255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:18.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:18 smithi156 ceph-mon[65097]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:18.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:18 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3621145385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:19.963 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mgr 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mon 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all crash 2024-04-24T16:47:19.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[81602]: Upgrade: osd.0 is safe to restart 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mgr 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.965 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mon 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all crash 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-24T16:47:19.966 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:19 smithi070 ceph-mon[86496]: Upgrade: osd.0 is safe to restart 2024-04-24T16:47:20.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:20.325 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mgr 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mon 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:20.326 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:20.327 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all crash 2024-04-24T16:47:20.327 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:20.327 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-24T16:47:20.327 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-24T16:47:20.327 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:19 smithi156 ceph-mon[65097]: Upgrade: osd.0 is safe to restart 2024-04-24T16:47:21.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[86496]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:21.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:21.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:47:21.042 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:21.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[81602]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:21.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:21.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:47:21.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:20 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:21.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:20 smithi156 ceph-mon[65097]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:21.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:20 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:21.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:20 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T16:47:21.061 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:20 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:21.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:21 smithi070 ceph-mon[81602]: Upgrade: Updating osd.0 2024-04-24T16:47:21.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:21 smithi070 ceph-mon[81602]: Deploying daemon osd.0 on smithi070 2024-04-24T16:47:21.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:21 smithi070 ceph-mon[86496]: Upgrade: Updating osd.0 2024-04-24T16:47:21.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:21 smithi070 ceph-mon[86496]: Deploying daemon osd.0 on smithi070 2024-04-24T16:47:22.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:21 smithi156 ceph-mon[65097]: Upgrade: Updating osd.0 2024-04-24T16:47:22.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:21 smithi156 ceph-mon[65097]: Deploying daemon osd.0 on smithi070 2024-04-24T16:47:23.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:23 smithi070 ceph-mon[81602]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:23.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:23 smithi070 ceph-mon[86496]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:23 smithi156 ceph-mon[65097]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:47:23.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:23 smithi070 systemd[1]: Stopping Ceph osd.0 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:47:24.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:24 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:24.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:24 smithi070 ceph-mon[81602]: osd.0 marked itself down and dead 2024-04-24T16:47:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:24 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:24.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:24 smithi070 ceph-mon[86496]: osd.0 marked itself down and dead 2024-04-24T16:47:24.294 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[29173]: 2024-04-24T16:47:23.815+0000 7f9b89a0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:47:24.294 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[29173]: 2024-04-24T16:47:23.815+0000 7f9b89a0d700 -1 osd.0 112 *** Got signal Terminated *** 2024-04-24T16:47:24.294 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[29173]: 2024-04-24T16:47:23.815+0000 7f9b89a0d700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:47:24.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:24 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:24.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:24 smithi156 ceph-mon[65097]: osd.0 marked itself down and dead 2024-04-24T16:47:24.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:24 smithi070 podman[93334]: 2024-04-24 16:47:24.520832482 +0000 UTC m=+0.893530571 container died 5c223a67a3877bb934728b5632d24f4915f4981b8e20de290a2a1e2bfc424f43 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.openshift.expose-services=, release=754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, version=8, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, name=centos-stream, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870) 2024-04-24T16:47:25.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:25 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:47:25.292 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:25 smithi070 ceph-mon[86496]: osdmap e113: 8 total, 7 up, 8 in 2024-04-24T16:47:25.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:25 smithi070 ceph-mon[86496]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:25.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:25 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:47:25.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:25 smithi070 ceph-mon[81602]: osdmap e113: 8 total, 7 up, 8 in 2024-04-24T16:47:25.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:25 smithi070 ceph-mon[81602]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:25.293 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:24 smithi070 podman[93334]: 2024-04-24 16:47:24.966281092 +0000 UTC m=+1.338979187 container cleanup 5c223a67a3877bb934728b5632d24f4915f4981b8e20de290a2a1e2bfc424f43 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, ceph=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, distribution-scope=public, release=754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, RELEASE=HEAD, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0) 2024-04-24T16:47:25.293 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:24 smithi070 bash[93334]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0 2024-04-24T16:47:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:25 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:47:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:25 smithi156 ceph-mon[65097]: osdmap e113: 8 total, 7 up, 8 in 2024-04-24T16:47:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:25 smithi156 ceph-mon[65097]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:25.543 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:25 smithi070 podman[93352]: 2024-04-24 16:47:25.293115195 +0000 UTC m=+0.775693525 container remove 5c223a67a3877bb934728b5632d24f4915f4981b8e20de290a2a1e2bfc424f43 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, maintainer=Guillaume Abrioux , RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, release=754, GIT_BRANCH=HEAD) 2024-04-24T16:47:25.920 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:25 smithi070 podman[93448]: 2024-04-24 16:47:25.550472654 +0000 UTC m=+0.017790190 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:47:25.921 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:25 smithi070 podman[93448]: 2024-04-24 16:47:25.652209016 +0000 UTC m=+0.119526536 container create f96d05f9ed141d4763e337fc43875833802dca9e2a494fc31afa17febe724e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T16:47:25.921 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:25 smithi070 podman[93448]: 2024-04-24 16:47:25.846181742 +0000 UTC m=+0.313499265 container init f96d05f9ed141d4763e337fc43875833802dca9e2a494fc31afa17febe724e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424) 2024-04-24T16:47:25.921 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:25 smithi070 podman[93448]: 2024-04-24 16:47:25.849274024 +0000 UTC m=+0.316591552 container start f96d05f9ed141d4763e337fc43875833802dca9e2a494fc31afa17febe724e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-24T16:47:25.921 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:25 smithi070 podman[93448]: 2024-04-24 16:47:25.921119707 +0000 UTC m=+0.388437228 container attach f96d05f9ed141d4763e337fc43875833802dca9e2a494fc31afa17febe724e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:47:26.173 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:26 smithi070 ceph-mon[86496]: osdmap e114: 8 total, 7 up, 8 in 2024-04-24T16:47:26.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:26 smithi070 ceph-mon[81602]: osdmap e114: 8 total, 7 up, 8 in 2024-04-24T16:47:26.174 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:26 smithi070 podman[93448]: 2024-04-24 16:47:26.145939022 +0000 UTC m=+0.613256546 container died f96d05f9ed141d4763e337fc43875833802dca9e2a494fc31afa17febe724e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-24T16:47:26.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:26 smithi156 ceph-mon[65097]: osdmap e114: 8 total, 7 up, 8 in 2024-04-24T16:47:26.693 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:47:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:47:26] "GET /metrics HTTP/1.1" 200 38018 "" "Prometheus/2.43.0" 2024-04-24T16:47:26.693 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:26 smithi070 podman[93474]: 2024-04-24 16:47:26.691077497 +0000 UTC m=+0.539739055 container remove f96d05f9ed141d4763e337fc43875833802dca9e2a494fc31afa17febe724e9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-24T16:47:27.043 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:26 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-24T16:47:27.043 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:26 smithi070 systemd[1]: Stopped Ceph osd.0 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:47:27.043 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:26 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.0.service: Consumed 13.894s CPU time. 2024-04-24T16:47:27.043 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:26 smithi070 systemd[1]: Starting Ceph osd.0 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:47:27.386 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-mon[81602]: pgmap v69: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:47:27.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-mon[86496]: pgmap v69: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:47:27.387 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 podman[93580]: 2024-04-24 16:47:27.026890953 +0000 UTC m=+0.018309888 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:47:27.387 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 podman[93580]: 2024-04-24 16:47:27.150991511 +0000 UTC m=+0.142410437 container create 143c52e05a820e77605f205147d69211e8660f817b9659649410012ecabb9bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-24T16:47:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:27 smithi156 ceph-mon[65097]: pgmap v69: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:47:27.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 podman[93580]: 2024-04-24 16:47:27.386767119 +0000 UTC m=+0.378186045 container init 143c52e05a820e77605f205147d69211e8660f817b9659649410012ecabb9bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:47:27.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 podman[93580]: 2024-04-24 16:47:27.390176908 +0000 UTC m=+0.381595831 container start 143c52e05a820e77605f205147d69211e8660f817b9659649410012ecabb9bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:47:27.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 podman[93580]: 2024-04-24 16:47:27.470036494 +0000 UTC m=+0.461455419 container attach 143c52e05a820e77605f205147d69211e8660f817b9659649410012ecabb9bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:47:28.185 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 bash[93580]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-24T16:47:28.186 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T16:47:28.187 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-24T16:47:28.187 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-24T16:47:28.187 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-24T16:47:28.187 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-24T16:47:28.187 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T16:47:28.187 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate[93595]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-24T16:47:28.187 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:27 smithi070 podman[93580]: 2024-04-24 16:47:27.887597606 +0000 UTC m=+0.879016524 container died 143c52e05a820e77605f205147d69211e8660f817b9659649410012ecabb9bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:47:28.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:28 smithi070 ceph-mon[86496]: pgmap v70: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:28.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:28 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 64/627 objects degraded (10.207%), 18 pgs degraded (PG_DEGRADED) 2024-04-24T16:47:28.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:28 smithi070 ceph-mon[81602]: pgmap v70: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:28.524 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:28 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 64/627 objects degraded (10.207%), 18 pgs degraded (PG_DEGRADED) 2024-04-24T16:47:28.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:28 smithi156 ceph-mon[65097]: pgmap v70: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:28.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:28 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 64/627 objects degraded (10.207%), 18 pgs degraded (PG_DEGRADED) 2024-04-24T16:47:28.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:28 smithi070 podman[93729]: 2024-04-24 16:47:28.524186799 +0000 UTC m=+0.631295086 container remove 143c52e05a820e77605f205147d69211e8660f817b9659649410012ecabb9bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-24T16:47:28.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:28 smithi070 podman[93793]: 2024-04-24 16:47:28.733509457 +0000 UTC m=+0.106949194 container create fab1f03082000ea2ad8b2caad26984eca37e05843a151cc8e0eb73e0bbce847b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-24T16:47:28.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:28 smithi070 podman[93793]: 2024-04-24 16:47:28.645111452 +0000 UTC m=+0.018551161 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:47:29.085 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:28 smithi070 podman[93793]: 2024-04-24 16:47:28.985613127 +0000 UTC m=+0.359052821 container init fab1f03082000ea2ad8b2caad26984eca37e05843a151cc8e0eb73e0bbce847b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-24T16:47:29.085 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:28 smithi070 podman[93793]: 2024-04-24 16:47:28.988630144 +0000 UTC m=+0.362069841 container start fab1f03082000ea2ad8b2caad26984eca37e05843a151cc8e0eb73e0bbce847b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T16:47:29.085 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:29 smithi070 ceph-osd[93813]: -- 172.21.15.70:0/3243622737 <== mon.1 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564338dad680 con 0x564338d77400 2024-04-24T16:47:29.529 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:29 smithi070 bash[93793]: fab1f03082000ea2ad8b2caad26984eca37e05843a151cc8e0eb73e0bbce847b 2024-04-24T16:47:29.529 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:29 smithi070 systemd[1]: Started Ceph osd.0 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:47:29.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:29 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[93809]: 2024-04-24T16:47:29.529+0000 7f5abc51b7c0 -1 Falling back to public interface 2024-04-24T16:47:30.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:30.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:30 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:30 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:30 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:30 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:30 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:31.135 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:31 smithi070 ceph-mon[81602]: pgmap v71: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:31.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:31 smithi156 ceph-mon[65097]: pgmap v71: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:31.469 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:31 smithi070 ceph-mon[86496]: pgmap v71: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:32.543 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[93809]: 2024-04-24T16:47:32.258+0000 7f5abc51b7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-24T16:47:32.543 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[93809]: 2024-04-24T16:47:32.281+0000 7f5abc51b7c0 -1 osd.0 112 log_to_monitors true 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[86496]: pgmap v72: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 64/627 objects degraded (10.207%) 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[86496]: from='osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[86496]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[81602]: pgmap v72: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 64/627 objects degraded (10.207%) 2024-04-24T16:47:33.093 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[81602]: from='osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:47:33.094 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-mon[81602]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:47:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:33 smithi156 ceph-mon[65097]: pgmap v72: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 64/627 objects degraded (10.207%) 2024-04-24T16:47:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:33 smithi156 ceph-mon[65097]: from='osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:47:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:33 smithi156 ceph-mon[65097]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T16:47:33.793 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:47:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[93809]: 2024-04-24T16:47:33.347+0000 7f5aa996b700 -1 osd.0 112 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:47:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T16:47:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: osdmap e115: 8 total, 7 up, 8 in 2024-04-24T16:47:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: from='osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:47:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:47:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:34 smithi156 ceph-mon[65097]: from='osd.0 ' entity='osd.0' 2024-04-24T16:47:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T16:47:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: osdmap e115: 8 total, 7 up, 8 in 2024-04-24T16:47:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: from='osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:47:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:47:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:34.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[86496]: from='osd.0 ' entity='osd.0' 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: osdmap e115: 8 total, 7 up, 8 in 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: from='osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:34.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:34.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:34 smithi070 ceph-mon[81602]: from='osd.0 ' entity='osd.0' 2024-04-24T16:47:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[86496]: pgmap v74: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:47:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[86496]: osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320] boot 2024-04-24T16:47:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[86496]: osdmap e116: 8 total, 8 up, 8 in 2024-04-24T16:47:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:47:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[81602]: pgmap v74: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[81602]: osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320] boot 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[81602]: osdmap e116: 8 total, 8 up, 8 in 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:35.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:35 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:35 smithi156 ceph-mon[65097]: pgmap v74: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:35 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:47:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:35 smithi156 ceph-mon[65097]: osd.0 [v2:172.21.15.70:6802/531715320,v1:172.21.15.70:6803/531715320] boot 2024-04-24T16:47:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:35 smithi156 ceph-mon[65097]: osdmap e116: 8 total, 8 up, 8 in 2024-04-24T16:47:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:35 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T16:47:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:35 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:35 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:36.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:36 smithi070 ceph-mon[86496]: osdmap e117: 8 total, 8 up, 8 in 2024-04-24T16:47:36.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:36 smithi070 ceph-mon[86496]: pgmap v77: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:36.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:36 smithi070 ceph-mon[81602]: osdmap e117: 8 total, 8 up, 8 in 2024-04-24T16:47:36.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:36 smithi070 ceph-mon[81602]: pgmap v77: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:36.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:36 smithi156 ceph-mon[65097]: osdmap e117: 8 total, 8 up, 8 in 2024-04-24T16:47:36.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:36 smithi156 ceph-mon[65097]: pgmap v77: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:47:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:47:36] "GET /metrics HTTP/1.1" 200 38152 "" "Prometheus/2.43.0" 2024-04-24T16:47:38.042 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:47:37 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:47:37.964+0000 7f22938a6700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-24T16:47:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:39.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: pgmap v78: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: pgmap v79: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 64/627 objects degraded (10.207%) 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:39.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:38 smithi156 ceph-mon[65097]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: pgmap v78: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: pgmap v79: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 64/627 objects degraded (10.207%) 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[81602]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:47:39.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: pgmap v78: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: pgmap v79: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 64/627 objects degraded (10.207%) 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:39.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:38 smithi070 ceph-mon[86496]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-24T16:47:40.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:39 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/627 objects degraded (10.207%), 18 pgs degraded) 2024-04-24T16:47:40.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:39 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/627 objects degraded (10.207%), 18 pgs degraded) 2024-04-24T16:47:40.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:39 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/627 objects degraded (10.207%), 18 pgs degraded) 2024-04-24T16:47:41.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:40 smithi070 ceph-mon[81602]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-04-24T16:47:41.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:40 smithi070 ceph-mon[86496]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-04-24T16:47:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:40 smithi156 ceph-mon[65097]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-04-24T16:47:43.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:42 smithi070 ceph-mon[81602]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-24T16:47:43.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:42 smithi070 ceph-mon[86496]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-24T16:47:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:42 smithi156 ceph-mon[65097]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-24T16:47:44.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:43 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:44.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:43 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:43 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:45.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:44 smithi070 ceph-mon[86496]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:45.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:44 smithi070 ceph-mon[81602]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:44 smithi156 ceph-mon[65097]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:47:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:47:46] "GET /metrics HTTP/1.1" 200 38152 "" "Prometheus/2.43.0" 2024-04-24T16:47:47.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:46 smithi070 ceph-mon[81602]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:47:47.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:46 smithi070 ceph-mon[86496]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:47:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:46 smithi156 ceph-mon[65097]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:47:48.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:48.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:48.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:48.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:48.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:48.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:48.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:48.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:47:48.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:48.935 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:47:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:49 smithi070 ceph-mon[86496]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:47:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:49 smithi070 ceph-mon[81602]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:47:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:49 smithi156 ceph-mon[65097]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:47:49.588 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (4m) 16s ago 26m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (4m) 2m ago 26m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 16s ago 25m 63.7M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (5m) 2m ago 30m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (15m) 16s ago 32m 504M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (2m) 16s ago 32m 45.8M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (3m) 2m ago 31m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (2m) 16s ago 31m 40.1M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (5m) 16s ago 27m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (5m) 2m ago 27m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (20s) 16s ago 30m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (30m) 16s ago 30m 57.9M 2992M 17.2.0 e1d6a67b021e 9715ccc21911 2024-04-24T16:47:49.589 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (29m) 16s ago 29m 52.2M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (29m) 16s ago 29m 53.5M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (29m) 2m ago 29m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (28m) 2m ago 28m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (28m) 2m ago 28m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (27m) 2m ago 27m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (4m) 2m ago 27m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (25m) 16s ago 25m 77.4M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:47:49.590 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (25m) 2m ago 25m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:47:49.987 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:47:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:47:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:47:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:47:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:47:49.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 1 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 6 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:47:49.988 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:47:50.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:50 smithi070 ceph-mon[86496]: from='client.54279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:50.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:50 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/476938364' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:50.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:50 smithi070 ceph-mon[81602]: from='client.54279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:50.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:50 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/476938364' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "1/8 daemons upgraded", 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:47:50.324 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:47:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:50 smithi156 ceph-mon[65097]: from='client.54279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:50 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/476938364' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:47:51.311 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:51 smithi156 ceph-mon[65097]: from='client.54285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:51.312 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:51 smithi156 ceph-mon[65097]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:51.312 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:51 smithi156 ceph-mon[65097]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:51.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:51 smithi070 ceph-mon[86496]: from='client.54285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:51.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:51 smithi070 ceph-mon[86496]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:51.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:51 smithi070 ceph-mon[86496]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:51 smithi070 ceph-mon[81602]: from='client.54285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:51 smithi070 ceph-mon[81602]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:51 smithi070 ceph-mon[81602]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:52 smithi156 ceph-mon[65097]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:52.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:52 smithi070 ceph-mon[86496]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:52.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:52 smithi070 ceph-mon[81602]: from='client.54303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:47:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:53 smithi156 ceph-mon[65097]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:53 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:53.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:53 smithi070 ceph-mon[81602]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:53.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:53 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:53 smithi070 ceph-mon[86496]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:53 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:54 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:54 smithi070 ceph-mon[86496]: Upgrade: osd.1 is safe to restart 2024-04-24T16:47:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:54 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:54 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:54 smithi070 ceph-mon[81602]: Upgrade: osd.1 is safe to restart 2024-04-24T16:47:54.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:54 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:54 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-24T16:47:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:54 smithi156 ceph-mon[65097]: Upgrade: osd.1 is safe to restart 2024-04-24T16:47:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:54 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:47:55.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[86496]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:55.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:55.230 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:47:55.231 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:55.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[81602]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:55.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:55.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:47:55.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:55 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:55 smithi156 ceph-mon[65097]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:47:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:55 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:47:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:55 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T16:47:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:55 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:47:56.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:56 smithi070 ceph-mon[86496]: Upgrade: Updating osd.1 2024-04-24T16:47:56.214 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:56 smithi070 ceph-mon[86496]: Deploying daemon osd.1 on smithi070 2024-04-24T16:47:56.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:56 smithi070 ceph-mon[81602]: Upgrade: Updating osd.1 2024-04-24T16:47:56.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:56 smithi070 ceph-mon[81602]: Deploying daemon osd.1 on smithi070 2024-04-24T16:47:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:56 smithi156 ceph-mon[65097]: Upgrade: Updating osd.1 2024-04-24T16:47:56.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:56 smithi156 ceph-mon[65097]: Deploying daemon osd.1 on smithi070 2024-04-24T16:47:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:47:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:47:56] "GET /metrics HTTP/1.1" 200 38167 "" "Prometheus/2.43.0" 2024-04-24T16:47:57.404 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:57 smithi070 ceph-mon[86496]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:57.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:57 smithi070 ceph-mon[81602]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:57 smithi156 ceph-mon[65097]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:58.037 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:57 smithi070 systemd[1]: Stopping Ceph osd.1 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:47:58.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:58 smithi070 ceph-mon[86496]: osd.1 marked itself down and dead 2024-04-24T16:47:58.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:58 smithi070 ceph-mon[81602]: osd.1 marked itself down and dead 2024-04-24T16:47:58.293 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[32158]: 2024-04-24T16:47:58.033+0000 7f2acbf93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:47:58.293 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[32158]: 2024-04-24T16:47:58.034+0000 7f2acbf93700 -1 osd.1 117 *** Got signal Terminated *** 2024-04-24T16:47:58.294 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:58 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[32158]: 2024-04-24T16:47:58.034+0000 7f2acbf93700 -1 osd.1 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:47:58.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:58 smithi156 ceph-mon[65097]: osd.1 marked itself down and dead 2024-04-24T16:47:58.905 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:58 smithi070 podman[98507]: 2024-04-24 16:47:58.604498093 +0000 UTC m=+0.706956892 container died 9715ccc219111393da5a0033c402b5849733fae99891f2806b2ffc88f2e67d42 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-type=git, release=754, io.openshift.tags=base centos centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., name=centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-04-24T16:47:59.193 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:59 smithi070 ceph-mon[86496]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:59.193 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:59 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:47:59.193 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:47:59 smithi070 ceph-mon[86496]: osdmap e118: 8 total, 7 up, 8 in 2024-04-24T16:47:59.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:59 smithi070 ceph-mon[81602]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:59.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:59 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:47:59.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:47:59 smithi070 ceph-mon[81602]: osdmap e118: 8 total, 7 up, 8 in 2024-04-24T16:47:59.194 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:58 smithi070 podman[98507]: 2024-04-24 16:47:58.908909027 +0000 UTC m=+1.011367832 container cleanup 9715ccc219111393da5a0033c402b5849733fae99891f2806b2ffc88f2e67d42 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, version=8, RELEASE=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , release=754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream) 2024-04-24T16:47:59.195 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:58 smithi070 bash[98507]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1 2024-04-24T16:47:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:59 smithi156 ceph-mon[65097]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:47:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:59 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:47:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:47:59 smithi156 ceph-mon[65097]: osdmap e118: 8 total, 7 up, 8 in 2024-04-24T16:47:59.527 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:59 smithi070 podman[98534]: 2024-04-24 16:47:59.193913041 +0000 UTC m=+0.585893604 container remove 9715ccc219111393da5a0033c402b5849733fae99891f2806b2ffc88f2e67d42 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, architecture=x86_64, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., ceph=True, release=754, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-04-24T16:47:59.793 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:59 smithi070 podman[98629]: 2024-04-24 16:47:59.528190279 +0000 UTC m=+0.109581508 container create 3ea28e3212f3b6f44653ffef5e1122a0ed84935ab6119154b1947738baa09fcd (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=) 2024-04-24T16:47:59.793 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:59 smithi070 podman[98629]: 2024-04-24 16:47:59.436708824 +0000 UTC m=+0.018100066 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:47:59.793 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:59 smithi070 podman[98629]: 2024-04-24 16:47:59.755609266 +0000 UTC m=+0.337000490 container init 3ea28e3212f3b6f44653ffef5e1122a0ed84935ab6119154b1947738baa09fcd (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0) 2024-04-24T16:47:59.793 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:59 smithi070 podman[98629]: 2024-04-24 16:47:59.759385618 +0000 UTC m=+0.340776844 container start 3ea28e3212f3b6f44653ffef5e1122a0ed84935ab6119154b1947738baa09fcd (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:48:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:00 smithi070 ceph-mon[81602]: osdmap e119: 8 total, 7 up, 8 in 2024-04-24T16:48:00.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:00 smithi070 ceph-mon[86496]: osdmap e119: 8 total, 7 up, 8 in 2024-04-24T16:48:00.294 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:47:59 smithi070 podman[98629]: 2024-04-24 16:47:59.864141886 +0000 UTC m=+0.445533120 container attach 3ea28e3212f3b6f44653ffef5e1122a0ed84935ab6119154b1947738baa09fcd (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:48:00.294 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:00 smithi070 podman[98629]: 2024-04-24 16:48:00.054413954 +0000 UTC m=+0.635805178 container died 3ea28e3212f3b6f44653ffef5e1122a0ed84935ab6119154b1947738baa09fcd (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-24T16:48:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:00 smithi156 ceph-mon[65097]: osdmap e119: 8 total, 7 up, 8 in 2024-04-24T16:48:00.600 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:00 smithi070 podman[98655]: 2024-04-24 16:48:00.584115984 +0000 UTC m=+0.524242767 container remove 3ea28e3212f3b6f44653ffef5e1122a0ed84935ab6119154b1947738baa09fcd (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-24T16:48:00.600 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:00 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-24T16:48:00.600 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:00 smithi070 systemd[1]: Stopped Ceph osd.1 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:48:01.020 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:00 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.1.service: Consumed 14.513s CPU time. 2024-04-24T16:48:01.020 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:00 smithi070 systemd[1]: Starting Ceph osd.1 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:48:01.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-mon[86496]: pgmap v92: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:48:01.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-mon[86496]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:48:01.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-mon[81602]: pgmap v92: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:48:01.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-mon[81602]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:48:01.295 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 podman[98760]: 2024-04-24 16:48:00.923728829 +0000 UTC m=+0.019053641 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:48:01.295 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 podman[98760]: 2024-04-24 16:48:01.026597314 +0000 UTC m=+0.121922115 container create deef0f07ac096c23458a72b653419ba1f5d2627a55c8031e1bbabd897bf5956e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-24T16:48:01.295 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 podman[98760]: 2024-04-24 16:48:01.254935651 +0000 UTC m=+0.350260458 container init deef0f07ac096c23458a72b653419ba1f5d2627a55c8031e1bbabd897bf5956e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-24T16:48:01.296 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 podman[98760]: 2024-04-24 16:48:01.258284533 +0000 UTC m=+0.353609344 container start deef0f07ac096c23458a72b653419ba1f5d2627a55c8031e1bbabd897bf5956e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-24T16:48:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:01 smithi156 ceph-mon[65097]: pgmap v92: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-24T16:48:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:01 smithi156 ceph-mon[65097]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:48:01.732 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 podman[98760]: 2024-04-24 16:48:01.363189559 +0000 UTC m=+0.458514371 container attach deef0f07ac096c23458a72b653419ba1f5d2627a55c8031e1bbabd897bf5956e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-24T16:48:02.043 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 bash[98760]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-24T16:48:02.044 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-24T16:48:02.045 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-24T16:48:02.045 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-24T16:48:02.045 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T16:48:02.045 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate[98775]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-24T16:48:02.045 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:01 smithi070 podman[98760]: 2024-04-24 16:48:01.756598892 +0000 UTC m=+0.851923695 container died deef0f07ac096c23458a72b653419ba1f5d2627a55c8031e1bbabd897bf5956e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T16:48:02.672 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 podman[98915]: 2024-04-24 16:48:02.367197458 +0000 UTC m=+0.601787889 container remove deef0f07ac096c23458a72b653419ba1f5d2627a55c8031e1bbabd897bf5956e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-24T16:48:02.672 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 podman[98978]: 2024-04-24 16:48:02.576284161 +0000 UTC m=+0.112596347 container create 3a6552201d2f7ec683d5088b15253229c0026e9bbd27f213a43603c5b57737b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-04-24T16:48:02.672 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 podman[98978]: 2024-04-24 16:48:02.482307035 +0000 UTC m=+0.018619234 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:48:03.055 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 podman[98978]: 2024-04-24 16:48:02.79601451 +0000 UTC m=+0.332326697 container init 3a6552201d2f7ec683d5088b15253229c0026e9bbd27f213a43603c5b57737b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T16:48:03.055 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 podman[98978]: 2024-04-24 16:48:02.799050184 +0000 UTC m=+0.335362371 container start 3a6552201d2f7ec683d5088b15253229c0026e9bbd27f213a43603c5b57737b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-04-24T16:48:03.055 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 ceph-osd[98997]: -- 172.21.15.70:0/2764857949 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573a4dc5680 con 0x5573a4d8f000 2024-04-24T16:48:03.056 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 bash[98978]: 3a6552201d2f7ec683d5088b15253229c0026e9bbd27f213a43603c5b57737b2 2024-04-24T16:48:03.056 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:02 smithi070 systemd[1]: Started Ceph osd.1 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:48:03.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[81602]: pgmap v93: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:48:03.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:03.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[86496]: pgmap v93: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:48:03.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:03 smithi156 ceph-mon[65097]: pgmap v93: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:48:03.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:03.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:03.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:03.758 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[98993]: 2024-04-24T16:48:03.340+0000 7faa319e17c0 -1 Falling back to public interface 2024-04-24T16:48:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:04 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:04.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:04 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:04.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:04 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:05 smithi070 ceph-mon[86496]: pgmap v94: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:05 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 87/627 objects degraded (13.876%), 23 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:05 smithi070 ceph-mon[86496]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-24T16:48:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:05 smithi070 ceph-mon[81602]: pgmap v94: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:05 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 87/627 objects degraded (13.876%), 23 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:05 smithi070 ceph-mon[81602]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-24T16:48:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:05 smithi156 ceph-mon[65097]: pgmap v94: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:05 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 87/627 objects degraded (13.876%), 23 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:05 smithi156 ceph-mon[65097]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-24T16:48:06.293 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[98993]: 2024-04-24T16:48:06.046+0000 7faa319e17c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-24T16:48:06.293 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[98993]: 2024-04-24T16:48:06.078+0000 7faa319e17c0 -1 osd.1 117 log_to_monitors true 2024-04-24T16:48:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:48:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:48:06] "GET /metrics HTTP/1.1" 200 38239 "" "Prometheus/2.43.0" 2024-04-24T16:48:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:07 smithi070 ceph-mon[86496]: pgmap v95: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:07 smithi070 ceph-mon[86496]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:48:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:07 smithi070 ceph-mon[81602]: pgmap v95: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:07 smithi070 ceph-mon[81602]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:48:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:07 smithi156 ceph-mon[65097]: pgmap v95: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:07 smithi156 ceph-mon[65097]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T16:48:07.793 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:48:07 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[98993]: 2024-04-24T16:48:07.347+0000 7faa1ee31700 -1 osd.1 117 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:48:08.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[86496]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T16:48:08.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[86496]: osdmap e120: 8 total, 7 up, 8 in 2024-04-24T16:48:08.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[86496]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:48:08.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[86496]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' 2024-04-24T16:48:08.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:08.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:08.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[81602]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T16:48:08.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[81602]: osdmap e120: 8 total, 7 up, 8 in 2024-04-24T16:48:08.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[81602]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:48:08.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[81602]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' 2024-04-24T16:48:08.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:08.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:08 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:08 smithi156 ceph-mon[65097]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T16:48:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:08 smithi156 ceph-mon[65097]: osdmap e120: 8 total, 7 up, 8 in 2024-04-24T16:48:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:08 smithi156 ceph-mon[65097]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:48:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:08 smithi156 ceph-mon[65097]: from='osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580]' entity='osd.1' 2024-04-24T16:48:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:08 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:08 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:09.266 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[86496]: pgmap v97: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:09.267 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:48:09.267 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[86496]: osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580] boot 2024-04-24T16:48:09.267 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[86496]: osdmap e121: 8 total, 8 up, 8 in 2024-04-24T16:48:09.267 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:48:09.267 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:09.267 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:09.267 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[81602]: pgmap v97: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:09.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:48:09.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[81602]: osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580] boot 2024-04-24T16:48:09.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[81602]: osdmap e121: 8 total, 8 up, 8 in 2024-04-24T16:48:09.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:48:09.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:09.268 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:09 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:09 smithi156 ceph-mon[65097]: pgmap v97: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 87/627 objects degraded (13.876%) 2024-04-24T16:48:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:09 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:48:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:09 smithi156 ceph-mon[65097]: osd.1 [v2:172.21.15.70:6810/4008134580,v1:172.21.15.70:6811/4008134580] boot 2024-04-24T16:48:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:09 smithi156 ceph-mon[65097]: osdmap e121: 8 total, 8 up, 8 in 2024-04-24T16:48:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:09 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T16:48:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:09 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:09.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:09 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:10.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:10 smithi070 ceph-mon[81602]: osdmap e122: 8 total, 8 up, 8 in 2024-04-24T16:48:10.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:10 smithi070 ceph-mon[81602]: pgmap v100: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/627 objects degraded (13.876%) 2024-04-24T16:48:10.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:10 smithi070 ceph-mon[86496]: osdmap e122: 8 total, 8 up, 8 in 2024-04-24T16:48:10.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:10 smithi070 ceph-mon[86496]: pgmap v100: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/627 objects degraded (13.876%) 2024-04-24T16:48:10.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:10 smithi156 ceph-mon[65097]: osdmap e122: 8 total, 8 up, 8 in 2024-04-24T16:48:10.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:10 smithi156 ceph-mon[65097]: pgmap v100: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/627 objects degraded (13.876%) 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: pgmap v101: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/627 objects degraded (13.876%) 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:48:13.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: pgmap v101: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/627 objects degraded (13.876%) 2024-04-24T16:48:13.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:48:13.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:48:13.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:48:13.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:48:13.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:48:13.300 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.302 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: pgmap v101: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 87/627 objects degraded (13.876%) 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:13.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:13.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:13.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:13.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:14.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: pgmap v102: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: pgmap v103: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:48:14.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: Upgrade: Finalizing container_image settings 2024-04-24T16:48:14.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: Upgrade: Complete! 2024-04-24T16:48:14.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: pgmap v104: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.200 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: pgmap v105: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: pgmap v106: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: pgmap v107: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 16 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: pgmap v102: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: pgmap v103: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: Upgrade: Finalizing container_image settings 2024-04-24T16:48:14.201 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: Upgrade: Complete! 2024-04-24T16:48:14.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: pgmap v104: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: pgmap v105: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: pgmap v106: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: pgmap v107: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:14.202 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:13 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 16 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: pgmap v102: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: pgmap v103: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:48:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: Upgrade: Finalizing container_image settings 2024-04-24T16:48:14.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: Upgrade: Complete! 2024-04-24T16:48:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: pgmap v104: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: pgmap v105: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: pgmap v106: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: pgmap v107: 161 pgs: 16 active+undersized+degraded, 27 active+undersized, 118 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-04-24T16:48:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:13 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 55/627 objects degraded (8.772%), 16 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:15.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:14 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/627 objects degraded (8.772%), 16 pgs degraded) 2024-04-24T16:48:15.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:14 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/627 objects degraded (8.772%), 16 pgs degraded) 2024-04-24T16:48:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:14 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/627 objects degraded (8.772%), 16 pgs degraded) 2024-04-24T16:48:16.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:15 smithi070 ceph-mon[86496]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-24T16:48:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:15 smithi070 ceph-mon[81602]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-24T16:48:16.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:15 smithi156 ceph-mon[65097]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-24T16:48:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:48:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:48:16] "GET /metrics HTTP/1.1" 200 38239 "" "Prometheus/2.43.0" 2024-04-24T16:48:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:18 smithi156 ceph-mon[65097]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[86496]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[81602]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:18.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:18.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:18.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:20.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:20 smithi156 ceph-mon[65097]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:48:20.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:20 smithi070 ceph-mon[86496]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:48:20.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:20 smithi070 ceph-mon[81602]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-24T16:48:21.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:48:22.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:22 smithi156 ceph-mon[65097]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:22.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:22 smithi156 ceph-mon[65097]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:22.430 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:22 smithi070 ceph-mon[86496]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:22.430 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:22 smithi070 ceph-mon[86496]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:22.430 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:22 smithi070 ceph-mon[81602]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:22.430 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:22 smithi070 ceph-mon[81602]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:23.519 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:48:23.519 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (5m) 15s ago 27m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:48:23.519 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (4m) 2m ago 26m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:48:23.519 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 15s ago 26m 63.8M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:48:23.519 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (6m) 2m ago 31m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:48:23.519 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (16m) 15s ago 32m 506M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (3m) 15s ago 33m 46.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (3m) 2m ago 31m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (2m) 15s ago 31m 41.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (5m) 15s ago 27m 16.1M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (5m) 2m ago 27m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (54s) 15s ago 31m 50.4M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (21s) 15s ago 30m 14.8M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (30m) 15s ago 30m 52.3M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (29m) 15s ago 29m 54.0M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (29m) 2m ago 29m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (29m) 2m ago 29m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (28m) 2m ago 28m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (28m) 2m ago 28m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:48:23.520 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (5m) 2m ago 27m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:48:23.521 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (26m) 15s ago 26m 77.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:48:23.521 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (26m) 2m ago 26m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:48:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:23 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:23 smithi156 ceph-mon[65097]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:48:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:23 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:23.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:23 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:23.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:23 smithi070 ceph-mon[86496]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:48:23.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:23 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:23.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:23 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:23.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:23 smithi070 ceph-mon[81602]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:48:23.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:23 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:24.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-04-24T16:48:24.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:24 smithi070 ceph-mon[86496]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:24.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:24 smithi070 ceph-mon[81602]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:24.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:24 smithi156 ceph-mon[65097]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:25.596 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:25 smithi070 ceph-mon[86496]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:25.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:25 smithi070 ceph-mon[81602]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:25.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:25 smithi156 ceph-mon[65097]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:26.162 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:48:26.420 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:26 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/563574886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:26.420 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:26 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/563574886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:26.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:26 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/563574886' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:26.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:48:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:48:26] "GET /metrics HTTP/1.1" 200 38242 "" "Prometheus/2.43.0" 2024-04-24T16:48:26.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-04-24T16:48:27.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:27 smithi156 ceph-mon[65097]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:27.774 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:27 smithi070 ceph-mon[86496]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:27.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:27 smithi070 ceph-mon[81602]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:29 smithi070 ceph-mon[81602]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:29 smithi070 ceph-mon[81602]: from='client.54333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:30.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:29 smithi070 ceph-mon[86496]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:30.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:29 smithi070 ceph-mon[86496]: from='client.54333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:30.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:29 smithi156 ceph-mon[65097]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:30.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:29 smithi156 ceph-mon[65097]: from='client.54333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:31.264 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:48:31.904 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:31 smithi070 ceph-mon[86496]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:31.904 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:31 smithi070 ceph-mon[81602]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:31.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:48:32.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:31 smithi156 ceph-mon[65097]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:33.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:33.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:33.840 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": null, 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": false, 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "", 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": null, 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:48:33.841 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:48:34.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:33 smithi070 ceph-mon[81602]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:34.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:33 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:33 smithi070 ceph-mon[86496]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:33 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:33 smithi156 ceph-mon[65097]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:33 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:34.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:48:35.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:34 smithi070 ceph-mon[86496]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:35.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:34 smithi070 ceph-mon[81602]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:35.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:34 smithi156 ceph-mon[65097]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:36.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:35 smithi070 ceph-mon[86496]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:36.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:35 smithi070 ceph-mon[81602]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:36.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:35 smithi156 ceph-mon[65097]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:36.660 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:48:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:48:36] "GET /metrics HTTP/1.1" 200 38237 "" "Prometheus/2.43.0" 2024-04-24T16:48:36.660 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:48:36.660 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:48:36.660 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:48:36.960 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:36 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1529237686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:48:36.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:36 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1529237686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:48:37.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:36 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1529237686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:48:37.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-04-24T16:48:37.914 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:37 smithi070 ceph-mon[86496]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:37.914 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:37 smithi070 ceph-mon[81602]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:38.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:37 smithi156 ceph-mon[65097]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:40.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:39 smithi156 ceph-mon[65097]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:40.217 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:39 smithi070 ceph-mon[81602]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:40.217 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:39 smithi070 ceph-mon[86496]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:48:41.148 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:40 smithi070 ceph-mon[86496]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:41.148 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:40 smithi070 ceph-mon[81602]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:41.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:40 smithi156 ceph-mon[65097]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:41.398 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:48:42.036 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:41 smithi070 ceph-mon[86496]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:42.037 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:42.037 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:42.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:41 smithi070 ceph-mon[81602]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:42.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:42.037 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:42.059 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-24T16:48:42.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:41 smithi156 ceph-mon[65097]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:48:42.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:42.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[81602]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[81602]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[81602]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[81602]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:48:43.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[86496]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:48:43.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:43.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:43.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[86496]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-04-24T16:48:43.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[86496]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:43.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:43.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:42 smithi070 ceph-mon[86496]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:48:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:42 smithi156 ceph-mon[65097]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:48:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:48:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:42 smithi156 ceph-mon[65097]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-04-24T16:48:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:42 smithi156 ceph-mon[65097]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:42 smithi156 ceph-mon[65097]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:48:44.155 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (5m) 37s ago 27m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (4m) 3m ago 27m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 37s ago 26m 63.8M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (6m) 3m ago 31m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (16m) 37s ago 33m 506M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (3m) 37s ago 33m 46.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (4m) 3m ago 32m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:48:44.864 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (2m) 37s ago 32m 41.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (6m) 37s ago 28m 16.1M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (6m) 3m ago 28m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (76s) 37s ago 31m 50.4M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (42s) 37s ago 31m 14.8M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (30m) 37s ago 30m 52.3M 2992M 17.2.0 e1d6a67b021e cd0cd7a00636 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (30m) 37s ago 30m 54.0M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (29m) 3m ago 29m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (29m) 3m ago 29m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (29m) 3m ago 29m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (28m) 3m ago 28m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (5m) 3m ago 28m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:48:44.865 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (26m) 37s ago 26m 77.5M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:48:44.866 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (26m) 3m ago 26m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:48:45.296 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:48:45.296 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:48:45.296 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:48:45.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:48:45.296 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:48:45.296 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:48:45.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 7 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:48:45.297 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:48:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:45 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:45 smithi156 ceph-mon[65097]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:45 smithi156 ceph-mon[65097]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:45.417 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:45 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:45.417 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:45 smithi070 ceph-mon[86496]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:45.417 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:45 smithi070 ceph-mon[86496]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:45.417 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:45 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:45.417 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:45 smithi070 ceph-mon[81602]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:45.417 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:45 smithi070 ceph-mon[81602]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:45.636 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:48:45.636 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", 2024-04-24T16:48:45.636 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:48:45.636 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-04-24T16:48:45.636 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:48:45.636 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "", 2024-04-24T16:48:45.636 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc image", 2024-04-24T16:48:45.637 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:48:45.637 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='client.54363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='client.54369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2247069557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.403 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='client.54363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='client.54369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2247069557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.404 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-24T16:48:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='client.54363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='client.54369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2247069557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:48:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-24T16:48:46.713 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:48:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:48:46] "GET /metrics HTTP/1.1" 200 38237 "" "Prometheus/2.43.0" 2024-04-24T16:48:47.384 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mgr 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mon 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all crash 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: Upgrade: osd.2 is safe to restart 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:47.385 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mgr 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mon 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all crash 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: Upgrade: osd.2 is safe to restart 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:47.386 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:48:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:48:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:48:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mgr 2024-04-24T16:48:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mon 2024-04-24T16:48:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all crash 2024-04-24T16:48:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-24T16:48:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: Upgrade: osd.2 is safe to restart 2024-04-24T16:48:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:48:48.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[86496]: Upgrade: Updating osd.2 2024-04-24T16:48:48.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:48.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:48:48.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[86496]: Deploying daemon osd.2 on smithi070 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[86496]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[81602]: Upgrade: Updating osd.2 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[81602]: Deploying daemon osd.2 on smithi070 2024-04-24T16:48:48.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:48 smithi070 ceph-mon[81602]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:48.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:48 smithi156 ceph-mon[65097]: Upgrade: Updating osd.2 2024-04-24T16:48:48.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:48.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T16:48:48.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:48:48.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:48 smithi156 ceph-mon[65097]: Deploying daemon osd.2 on smithi070 2024-04-24T16:48:48.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:48 smithi156 ceph-mon[65097]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:50.794 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:50 smithi070 systemd[1]: Stopping Ceph osd.2 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:48:51.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[35098]: 2024-04-24T16:48:50.804+0000 7fc611926700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:48:51.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[35098]: 2024-04-24T16:48:50.804+0000 7fc611926700 -1 osd.2 122 *** Got signal Terminated *** 2024-04-24T16:48:51.128 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[35098]: 2024-04-24T16:48:50.804+0000 7fc611926700 -1 osd.2 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:48:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:51 smithi156 ceph-mon[65097]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:51 smithi156 ceph-mon[65097]: osd.2 marked itself down and dead 2024-04-24T16:48:51.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:51 smithi070 ceph-mon[86496]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:51.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:51 smithi070 ceph-mon[86496]: osd.2 marked itself down and dead 2024-04-24T16:48:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:51 smithi070 ceph-mon[81602]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:48:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:51 smithi070 ceph-mon[81602]: osd.2 marked itself down and dead 2024-04-24T16:48:51.822 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:51 smithi070 podman[105011]: 2024-04-24 16:48:51.551141738 +0000 UTC m=+0.962261039 container died cd0cd7a0063617efd166bcb0d527ee26eb631715e806a645a86703eee51022cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.buildah.version=1.19.8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, version=8, ceph=True, architecture=x86_64) 2024-04-24T16:48:52.137 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:51 smithi070 podman[105011]: 2024-04-24 16:48:51.913469318 +0000 UTC m=+1.324588616 container cleanup cd0cd7a0063617efd166bcb0d527ee26eb631715e806a645a86703eee51022cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., version=8, GIT_CLEAN=True, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, maintainer=Guillaume Abrioux , RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, name=centos-stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870) 2024-04-24T16:48:52.137 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:51 smithi070 bash[105011]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2 2024-04-24T16:48:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:52 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:48:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:52 smithi156 ceph-mon[65097]: osdmap e123: 8 total, 7 up, 8 in 2024-04-24T16:48:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:52 smithi156 ceph-mon[65097]: osdmap e124: 8 total, 7 up, 8 in 2024-04-24T16:48:52.521 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:52 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:48:52.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:52 smithi070 ceph-mon[86496]: osdmap e123: 8 total, 7 up, 8 in 2024-04-24T16:48:52.522 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:52 smithi070 ceph-mon[86496]: osdmap e124: 8 total, 7 up, 8 in 2024-04-24T16:48:52.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:52 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:48:52.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:52 smithi070 ceph-mon[81602]: osdmap e123: 8 total, 7 up, 8 in 2024-04-24T16:48:52.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:52 smithi070 ceph-mon[81602]: osdmap e124: 8 total, 7 up, 8 in 2024-04-24T16:48:52.522 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:52 smithi070 podman[105030]: 2024-04-24 16:48:52.210228885 +0000 UTC m=+0.656999755 container remove cd0cd7a0063617efd166bcb0d527ee26eb631715e806a645a86703eee51022cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, release=754, vcs-type=git, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True, GIT_CLEAN=True, vendor=Red Hat, Inc., io.buildah.version=1.19.8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-24T16:48:52.793 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:52 smithi070 podman[105125]: 2024-04-24 16:48:52.424995645 +0000 UTC m=+0.018194258 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:48:52.793 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:52 smithi070 podman[105125]: 2024-04-24 16:48:52.566732999 +0000 UTC m=+0.159931596 container create 28cda0eb3290b9eec370703399f488fa61c9fe23765340649ee4279289b05ea1 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:48:53.095 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:52 smithi070 podman[105125]: 2024-04-24 16:48:52.8025643 +0000 UTC m=+0.395762897 container init 28cda0eb3290b9eec370703399f488fa61c9fe23765340649ee4279289b05ea1 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:48:53.095 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:52 smithi070 podman[105125]: 2024-04-24 16:48:52.805928764 +0000 UTC m=+0.399127374 container start 28cda0eb3290b9eec370703399f488fa61c9fe23765340649ee4279289b05ea1 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-24T16:48:53.096 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:52 smithi070 podman[105125]: 2024-04-24 16:48:52.87754236 +0000 UTC m=+0.470740958 container attach 28cda0eb3290b9eec370703399f488fa61c9fe23765340649ee4279289b05ea1 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-24T16:48:53.380 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:53 smithi070 ceph-mon[86496]: pgmap v129: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:48:53.380 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:53 smithi070 podman[105125]: 2024-04-24 16:48:53.096851542 +0000 UTC m=+0.690050131 container died 28cda0eb3290b9eec370703399f488fa61c9fe23765340649ee4279289b05ea1 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-04-24T16:48:53.381 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:53 smithi070 ceph-mon[81602]: pgmap v129: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:48:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:53 smithi156 ceph-mon[65097]: pgmap v129: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:48:53.631 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:53 smithi070 podman[105151]: 2024-04-24 16:48:53.614536924 +0000 UTC m=+0.508596504 container remove 28cda0eb3290b9eec370703399f488fa61c9fe23765340649ee4279289b05ea1 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:48:54.043 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:53 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-24T16:48:54.043 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:53 smithi070 systemd[1]: Stopped Ceph osd.2 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:48:54.043 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:53 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.2.service: Consumed 12.125s CPU time. 2024-04-24T16:48:54.043 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:53 smithi070 systemd[1]: Starting Ceph osd.2 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:48:54.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:54.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-mon[81602]: pgmap v131: 161 pgs: 1 activating+undersized, 13 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:54.360 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 57/627 objects degraded (9.091%), 13 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:54.360 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:54.360 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-mon[86496]: pgmap v131: 161 pgs: 1 activating+undersized, 13 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:54.361 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 57/627 objects degraded (9.091%), 13 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:54.361 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 podman[105257]: 2024-04-24 16:48:53.965701116 +0000 UTC m=+0.018036421 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:48:54.361 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 podman[105257]: 2024-04-24 16:48:54.098880859 +0000 UTC m=+0.151216173 container create 288cb49301857dc73b0ac1b0d8929af189632c3a613811dd631b811726a69456 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:48:54.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:54 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:48:54.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:54 smithi156 ceph-mon[65097]: pgmap v131: 161 pgs: 1 activating+undersized, 13 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:54.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:54 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 57/627 objects degraded (9.091%), 13 pgs degraded (PG_DEGRADED) 2024-04-24T16:48:54.704 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 podman[105257]: 2024-04-24 16:48:54.360609532 +0000 UTC m=+0.412944838 container init 288cb49301857dc73b0ac1b0d8929af189632c3a613811dd631b811726a69456 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-24T16:48:54.704 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 podman[105257]: 2024-04-24 16:48:54.36397362 +0000 UTC m=+0.416308934 container start 288cb49301857dc73b0ac1b0d8929af189632c3a613811dd631b811726a69456 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-24T16:48:54.704 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 podman[105257]: 2024-04-24 16:48:54.44401737 +0000 UTC m=+0.496352680 container attach 288cb49301857dc73b0ac1b0d8929af189632c3a613811dd631b811726a69456 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-24T16:48:55.043 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:55.043 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:55.043 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 bash[105257]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-24T16:48:55.044 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-24T16:48:55.045 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T16:48:55.045 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate[105271]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-24T16:48:55.045 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:54 smithi070 podman[105257]: 2024-04-24 16:48:54.863505226 +0000 UTC m=+0.915840523 container died 288cb49301857dc73b0ac1b0d8929af189632c3a613811dd631b811726a69456 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:48:55.793 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:55 smithi070 podman[105402]: 2024-04-24 16:48:55.481397545 +0000 UTC m=+0.612748129 container remove 288cb49301857dc73b0ac1b0d8929af189632c3a613811dd631b811726a69456 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-24T16:48:55.793 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:55 smithi070 podman[105460]: 2024-04-24 16:48:55.594118877 +0000 UTC m=+0.017855918 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:48:55.793 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:55 smithi070 podman[105460]: 2024-04-24 16:48:55.699104296 +0000 UTC m=+0.122841378 container create 161808df2d438da4491f50c061f2211ed4d93331b1b32e31bfc9f400ce216375 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-24T16:48:56.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:55 smithi070 podman[105460]: 2024-04-24 16:48:55.927031924 +0000 UTC m=+0.350768967 container init 161808df2d438da4491f50c061f2211ed4d93331b1b32e31bfc9f400ce216375 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:48:56.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:55 smithi070 podman[105460]: 2024-04-24 16:48:55.93015395 +0000 UTC m=+0.353890993 container start 161808df2d438da4491f50c061f2211ed4d93331b1b32e31bfc9f400ce216375 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:48:56.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:55 smithi070 ceph-osd[105480]: -- 172.21.15.70:0/2737160147 <== mon.1 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557af4a2f680 con 0x557af49f9400 2024-04-24T16:48:56.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:56 smithi070 bash[105460]: 161808df2d438da4491f50c061f2211ed4d93331b1b32e31bfc9f400ce216375 2024-04-24T16:48:56.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:56 smithi070 systemd[1]: Started Ceph osd.2 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:48:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:48:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:48:56] "GET /metrics HTTP/1.1" 200 38242 "" "Prometheus/2.43.0" 2024-04-24T16:48:56.793 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[105476]: 2024-04-24T16:48:56.480+0000 7f3f4f5c07c0 -1 Falling back to public interface 2024-04-24T16:48:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[86496]: pgmap v132: 161 pgs: 1 activating+undersized, 13 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[81602]: pgmap v132: 161 pgs: 1 activating+undersized, 13 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:57 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:57 smithi156 ceph-mon[65097]: pgmap v132: 161 pgs: 1 activating+undersized, 13 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:57 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:57 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:57 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:48:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:57 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:48:59.339 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:48:59 smithi070 ceph-mon[86496]: pgmap v133: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:59.339 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:48:59 smithi070 ceph-mon[81602]: pgmap v133: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:48:59.340 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[105476]: 2024-04-24T16:48:59.227+0000 7f3f4f5c07c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-24T16:48:59.340 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:48:59 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[105476]: 2024-04-24T16:48:59.251+0000 7f3f4f5c07c0 -1 osd.2 122 log_to_monitors true 2024-04-24T16:48:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:48:59 smithi156 ceph-mon[65097]: pgmap v133: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:49:00.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:00 smithi070 ceph-mon[81602]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:49:00.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:00 smithi070 ceph-mon[86496]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:49:00.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:49:00 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[105476]: 2024-04-24T16:49:00.102+0000 7f3f40a18700 -1 osd.2 122 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:49:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:00 smithi156 ceph-mon[65097]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T16:49:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:01 smithi156 ceph-mon[65097]: pgmap v134: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:49:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:01 smithi156 ceph-mon[65097]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T16:49:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:01 smithi156 ceph-mon[65097]: osdmap e125: 8 total, 7 up, 8 in 2024-04-24T16:49:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:01 smithi156 ceph-mon[65097]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:01.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[86496]: pgmap v134: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:49:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[86496]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T16:49:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[86496]: osdmap e125: 8 total, 7 up, 8 in 2024-04-24T16:49:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[86496]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[81602]: pgmap v134: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:49:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[81602]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T16:49:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[81602]: osdmap e125: 8 total, 7 up, 8 in 2024-04-24T16:49:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[81602]: from='osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974] boot 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: osdmap e126: 8 total, 8 up, 8 in 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:02.351 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[86496]: osdmap e127: 8 total, 8 up, 8 in 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974] boot 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: osdmap e126: 8 total, 8 up, 8 in 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:02.352 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:02 smithi070 ceph-mon[81602]: osdmap e127: 8 total, 8 up, 8 in 2024-04-24T16:49:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:49:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: osd.2 [v2:172.21.15.70:6818/1450138974,v1:172.21.15.70:6819/1450138974] boot 2024-04-24T16:49:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: osdmap e126: 8 total, 8 up, 8 in 2024-04-24T16:49:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T16:49:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:02 smithi156 ceph-mon[65097]: osdmap e127: 8 total, 8 up, 8 in 2024-04-24T16:49:03.356 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:03 smithi070 ceph-mon[86496]: pgmap v137: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:49:03.356 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:03 smithi070 ceph-mon[81602]: pgmap v137: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:49:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:03 smithi156 ceph-mon[65097]: pgmap v137: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-04-24T16:49:04.395 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:04 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded) 2024-04-24T16:49:04.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:04 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded) 2024-04-24T16:49:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:04 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/627 objects degraded (9.091%), 14 pgs degraded) 2024-04-24T16:49:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:05 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:05 smithi156 ceph-mon[65097]: pgmap v139: 161 pgs: 13 active+undersized, 148 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:49:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:05 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:05.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:05 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:05.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:05 smithi070 ceph-mon[86496]: pgmap v139: 161 pgs: 13 active+undersized, 148 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:49:05.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:05 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:05 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:05 smithi070 ceph-mon[81602]: pgmap v139: 161 pgs: 13 active+undersized, 148 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:49:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:05 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:49:06.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:49:06.522 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:49:06.524 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.525 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.526 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.527 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:06.528 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:49:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:49:06] "GET /metrics HTTP/1.1" 200 38109 "" "Prometheus/2.43.0" 2024-04-24T16:49:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-04-24T16:49:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-04-24T16:49:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:49:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: Upgrade: Finalizing container_image settings 2024-04-24T16:49:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: Upgrade: Complete! 2024-04-24T16:49:07.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:07.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:49:07.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:49:07.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:07 smithi156 ceph-mon[65097]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:49:07.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-04-24T16:49:07.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-04-24T16:49:07.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: Upgrade: Finalizing container_image settings 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: Upgrade: Complete! 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[81602]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: Upgrade: Finalizing container_image settings 2024-04-24T16:49:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: Upgrade: Complete! 2024-04-24T16:49:07.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:07.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:49:07.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:49:07.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:07 smithi070 ceph-mon[86496]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:49:08.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:08 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:08.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:08 smithi156 ceph-mon[65097]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:49:08.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:08 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:08.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:08 smithi070 ceph-mon[86496]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:49:08.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:08 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:08.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:08 smithi070 ceph-mon[81602]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:49:10.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:10 smithi070 ceph-mon[86496]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:10.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:10 smithi070 ceph-mon[81602]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:10.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:10 smithi156 ceph-mon[65097]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:12.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:12 smithi070 ceph-mon[86496]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:49:12.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:12.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:12 smithi070 ceph-mon[81602]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:49:12.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:12.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:12 smithi156 ceph-mon[65097]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:49:12.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:14.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:14 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:14.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:14 smithi070 ceph-mon[86496]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:14.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:14 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:14.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:14 smithi070 ceph-mon[81602]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:14.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:14 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:14.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:14 smithi156 ceph-mon[65097]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:16.543 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:49:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:49:16] "GET /metrics HTTP/1.1" 200 38109 "" "Prometheus/2.43.0" 2024-04-24T16:49:16.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:16 smithi070 ceph-mon[86496]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:49:16.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:16 smithi070 ceph-mon[86496]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:16.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:16 smithi070 ceph-mon[81602]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:49:16.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:16 smithi070 ceph-mon[81602]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:16.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:49:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:16 smithi156 ceph-mon[65097]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:49:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:16 smithi156 ceph-mon[65097]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:18.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:18.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:18.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:18.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (5m) 18s ago 28m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (5m) 3m ago 27m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 18s ago 27m 63.9M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (7m) 3m ago 32m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (17m) 18s ago 33m 508M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (3m) 18s ago 34m 49.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (4m) 3m ago 32m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (3m) 18s ago 32m 44.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:49:18.746 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (6m) 18s ago 28m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (6m) 3m ago 28m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (110s) 18s ago 32m 52.2M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (76s) 18s ago 31m 54.0M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (23s) 18s ago 31m 15.2M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (30m) 18s ago 30m 54.6M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (30m) 3m ago 30m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (30m) 3m ago 30m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (29m) 3m ago 29m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (29m) 3m ago 29m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (6m) 3m ago 28m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (27m) 18s ago 27m 77.7M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:49:18.747 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (27m) 3m ago 27m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:49:19.393 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:19 smithi070 ceph-mon[81602]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:19.393 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:19 smithi070 ceph-mon[86496]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:19 smithi156 ceph-mon[65097]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:19.416 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-04-24T16:49:20.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:20 smithi070 ceph-mon[81602]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:20 smithi070 ceph-mon[86496]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:20.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:20 smithi156 ceph-mon[65097]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:21.391 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:49:21.391 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:21 smithi070 ceph-mon[86496]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:21.392 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:21 smithi070 ceph-mon[81602]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:21.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:21 smithi156 ceph-mon[65097]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:22.249 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:22 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/204578591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:22.250 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:22 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/204578591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:22.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-04-24T16:49:22.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:22 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/204578591' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:23.196 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:23 smithi070 ceph-mon[86496]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:23.196 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:23 smithi070 ceph-mon[81602]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:23 smithi156 ceph-mon[65097]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:25.157 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:25 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:25.157 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:25 smithi070 ceph-mon[86496]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:25.157 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:25 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:25.158 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:25 smithi070 ceph-mon[81602]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:25 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:25 smithi156 ceph-mon[65097]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:26.336 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:49:26.336 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:26 smithi070 ceph-mon[86496]: from='client.54411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:26.336 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:26 smithi070 ceph-mon[81602]: from='client.54411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:26.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:26 smithi156 ceph-mon[65097]: from='client.54411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:26.625 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:49:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:49:26] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.43.0" 2024-04-24T16:49:26.925 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:49:27.229 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:27 smithi070 ceph-mon[86496]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:27.229 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:27 smithi070 ceph-mon[81602]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:27 smithi156 ceph-mon[65097]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:28.959 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:49:28.959 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": null, 2024-04-24T16:49:28.959 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": false, 2024-04-24T16:49:28.959 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "", 2024-04-24T16:49:28.960 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:49:28.960 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": null, 2024-04-24T16:49:28.960 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-04-24T16:49:28.960 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:49:28.960 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:49:29.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:29 smithi070 ceph-mon[86496]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:29.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:29 smithi070 ceph-mon[81602]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:29.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:29 smithi156 ceph-mon[65097]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:29.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:49:30.213 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:30 smithi070 ceph-mon[86496]: from='client.54417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:30.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:30 smithi070 ceph-mon[81602]: from='client.54417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:30.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:30 smithi156 ceph-mon[65097]: from='client.54417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:31.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:31 smithi156 ceph-mon[65097]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:31.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:31 smithi070 ceph-mon[86496]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:31.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:31 smithi070 ceph-mon[81602]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:31.597 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:49:31.597 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:49:31.597 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:49:32.183 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:32 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1355506631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:49:32.184 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:32.184 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:32 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1355506631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:49:32.184 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:32.206 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-04-24T16:49:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:32 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1355506631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:49:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:33 smithi156 ceph-mon[65097]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:33.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:33 smithi070 ceph-mon[86496]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:33.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:33 smithi070 ceph-mon[81602]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:35 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:35 smithi156 ceph-mon[65097]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:35.464 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:35 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:35.464 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:35 smithi070 ceph-mon[86496]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:35.464 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:35 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:35.465 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:35 smithi070 ceph-mon[81602]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:49:36.138 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:36 smithi070 ceph-mon[86496]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:36.138 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:36 smithi070 ceph-mon[81602]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:36.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:36 smithi156 ceph-mon[65097]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:49:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:49:36] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.43.0" 2024-04-24T16:49:36.995 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:49:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[86496]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:37.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[81602]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:37 smithi156 ceph-mon[65097]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:49:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:49:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:37.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-24T16:49:38.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[81602]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:49:38.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[81602]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-04-24T16:49:38.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[81602]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:38.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[81602]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:49:38.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[86496]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:49:38.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[86496]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-04-24T16:49:38.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[86496]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:38.255 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:38 smithi070 ceph-mon[86496]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:49:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:38 smithi156 ceph-mon[65097]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:49:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:38 smithi156 ceph-mon[65097]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-04-24T16:49:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:38 smithi156 ceph-mon[65097]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:38 smithi156 ceph-mon[65097]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:49:40.406 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:40 smithi070 ceph-mon[86496]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:40.406 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:40 smithi070 ceph-mon[81602]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:40.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:40 smithi156 ceph-mon[65097]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:40.534 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (6m) 40s ago 28m 26.0M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (5m) 4m ago 28m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 40s ago 27m 63.9M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (7m) 4m ago 32m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (17m) 40s ago 34m 508M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (4m) 40s ago 34m 49.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (5m) 4m ago 32m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (3m) 40s ago 33m 44.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:49:41.228 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (7m) 40s ago 29m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (7m) 4m ago 28m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (2m) 40s ago 32m 52.2M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (98s) 40s ago 32m 54.0M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (45s) 40s ago 31m 15.2M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (31m) 40s ago 31m 54.6M 2992M 17.2.0 e1d6a67b021e 787e5b0c4d2c 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (30m) 4m ago 30m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (30m) 4m ago 30m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (30m) 4m ago 30m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (29m) 4m ago 29m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (6m) 4m ago 29m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (27m) 40s ago 27m 77.7M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:49:41.229 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (27m) 4m ago 27m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:49:41.646 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:49:41.647 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:49:41.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:49:41.648 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:49:41.648 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-24T16:49:41.648 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 8 2024-04-24T16:49:41.648 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:49:41.648 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:49:41.998 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:49:41.998 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:49:41.998 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:49:41.999 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-24T16:49:41.999 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:49:41.999 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "3/8 daemons upgraded", 2024-04-24T16:49:41.999 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-04-24T16:49:41.999 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:49:41.999 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:49:42.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='client.54435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:42.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:49:42.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:49:42.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:42.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mgr 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mon 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all crash 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: Upgrade: osd.3 is safe to restart 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='client.54441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:42.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2609850334' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='client.54435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mgr 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mon 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all crash 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: Upgrade: osd.3 is safe to restart 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='client.54441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:42.046 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:41 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2609850334' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='client.54435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mgr 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mon 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:42.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all crash 2024-04-24T16:49:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-24T16:49:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-24T16:49:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: Upgrade: osd.3 is safe to restart 2024-04-24T16:49:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='client.54441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:42.162 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:41 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2609850334' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:49:43.114 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[86496]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:43.114 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[86496]: from='client.54459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:43.114 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:43.114 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:49:43.114 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:43.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[81602]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:43.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[81602]: from='client.54459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:43.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:43.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:49:43.115 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:42 smithi156 ceph-mon[65097]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:42 smithi156 ceph-mon[65097]: from='client.54459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:49:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T16:49:43.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:49:44.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:43 smithi070 ceph-mon[86496]: Upgrade: Updating osd.3 2024-04-24T16:49:44.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:43 smithi070 ceph-mon[86496]: Deploying daemon osd.3 on smithi070 2024-04-24T16:49:44.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:43 smithi070 ceph-mon[86496]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:44.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:43 smithi070 ceph-mon[81602]: Upgrade: Updating osd.3 2024-04-24T16:49:44.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:43 smithi070 ceph-mon[81602]: Deploying daemon osd.3 on smithi070 2024-04-24T16:49:44.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:43 smithi070 ceph-mon[81602]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:43 smithi156 ceph-mon[65097]: Upgrade: Updating osd.3 2024-04-24T16:49:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:43 smithi156 ceph-mon[65097]: Deploying daemon osd.3 on smithi070 2024-04-24T16:49:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:43 smithi156 ceph-mon[65097]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:44.950 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:44 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:44.950 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:44 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:45.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:44 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:45.833 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:45 smithi070 ceph-mon[81602]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:45.833 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:45 smithi070 ceph-mon[86496]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:45.833 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:45 smithi070 systemd[1]: Stopping Ceph osd.3 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:49:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:45 smithi156 ceph-mon[65097]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:49:46.293 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[38072]: 2024-04-24T16:49:45.869+0000 7f4786d2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:49:46.293 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[38072]: 2024-04-24T16:49:45.869+0000 7f4786d2d700 -1 osd.3 127 *** Got signal Terminated *** 2024-04-24T16:49:46.293 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[38072]: 2024-04-24T16:49:45.869+0000 7f4786d2d700 -1 osd.3 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:49:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:49:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:49:46] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.43.0" 2024-04-24T16:49:47.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:46 smithi156 ceph-mon[65097]: osd.3 marked itself down and dead 2024-04-24T16:49:47.238 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:46 smithi070 ceph-mon[81602]: osd.3 marked itself down and dead 2024-04-24T16:49:47.238 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:46 smithi070 ceph-mon[86496]: osd.3 marked itself down and dead 2024-04-24T16:49:47.543 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:47 smithi070 podman[111436]: 2024-04-24 16:49:47.23859281 +0000 UTC m=+1.489357385 container died 787e5b0c4d2ccca59a94044fed740234c114cbc8cc53096f2840169a4ce1dad2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, release=754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc.) 2024-04-24T16:49:48.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:49:48.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[86496]: osdmap e128: 8 total, 7 up, 8 in 2024-04-24T16:49:48.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[86496]: pgmap v168: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:49:48.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:48.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:48.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:49:48.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[81602]: osdmap e128: 8 total, 7 up, 8 in 2024-04-24T16:49:48.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[81602]: pgmap v168: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:49:48.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:48.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:48.044 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:47 smithi070 podman[111436]: 2024-04-24 16:49:47.56172715 +0000 UTC m=+1.812491725 container cleanup 787e5b0c4d2ccca59a94044fed740234c114cbc8cc53096f2840169a4ce1dad2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, ceph=True, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, name=centos-stream, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.buildah.version=1.19.8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8) 2024-04-24T16:49:48.044 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:47 smithi070 bash[111436]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3 2024-04-24T16:49:48.044 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:47 smithi070 podman[111464]: 2024-04-24 16:49:47.713168794 +0000 UTC m=+0.468456474 container remove 787e5b0c4d2ccca59a94044fed740234c114cbc8cc53096f2840169a4ce1dad2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.buildah.version=1.19.8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , architecture=x86_64, release=754, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, name=centos-stream, ceph=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=) 2024-04-24T16:49:48.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:47 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:49:48.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:47 smithi156 ceph-mon[65097]: osdmap e128: 8 total, 7 up, 8 in 2024-04-24T16:49:48.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:47 smithi156 ceph-mon[65097]: pgmap v168: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:49:48.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:48.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:49:48.466 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:48 smithi070 podman[111556]: 2024-04-24 16:49:48.035973093 +0000 UTC m=+0.018166639 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:49:48.467 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:48 smithi070 podman[111556]: 2024-04-24 16:49:48.173193988 +0000 UTC m=+0.155387572 container create be150a0f6175a0b57b3118cd3b266e28f2ff2e6e987e4463267ee2343c9bbe97 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-24T16:49:48.467 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:48 smithi070 podman[111556]: 2024-04-24 16:49:48.37564555 +0000 UTC m=+0.357839091 container init be150a0f6175a0b57b3118cd3b266e28f2ff2e6e987e4463267ee2343c9bbe97 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=) 2024-04-24T16:49:48.467 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:48 smithi070 podman[111556]: 2024-04-24 16:49:48.378707414 +0000 UTC m=+0.360900964 container start be150a0f6175a0b57b3118cd3b266e28f2ff2e6e987e4463267ee2343c9bbe97 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:49:48.467 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:48 smithi070 podman[111556]: 2024-04-24 16:49:48.467324334 +0000 UTC m=+0.449517889 container attach be150a0f6175a0b57b3118cd3b266e28f2ff2e6e987e4463267ee2343c9bbe97 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-24T16:49:48.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:48 smithi070 podman[111556]: 2024-04-24 16:49:48.668212145 +0000 UTC m=+0.650405697 container died be150a0f6175a0b57b3118cd3b266e28f2ff2e6e987e4463267ee2343c9bbe97 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:49:49.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:48 smithi156 ceph-mon[65097]: osdmap e129: 8 total, 7 up, 8 in 2024-04-24T16:49:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:48 smithi070 ceph-mon[81602]: osdmap e129: 8 total, 7 up, 8 in 2024-04-24T16:49:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:48 smithi070 ceph-mon[86496]: osdmap e129: 8 total, 7 up, 8 in 2024-04-24T16:49:49.772 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:49 smithi070 podman[111582]: 2024-04-24 16:49:49.337953994 +0000 UTC m=+0.664663792 container remove be150a0f6175a0b57b3118cd3b266e28f2ff2e6e987e4463267ee2343c9bbe97 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:49:49.772 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:49 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-24T16:49:49.772 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:49 smithi070 systemd[1]: Stopped Ceph osd.3 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:49:49.772 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:49 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.3.service: Consumed 16.802s CPU time. 2024-04-24T16:49:49.772 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:49 smithi070 systemd[1]: Starting Ceph osd.3 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:49:50.024 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:49 smithi070 ceph-mon[86496]: pgmap v170: 161 pgs: 17 active+undersized+degraded, 13 peering, 2 stale+active+clean, 29 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-04-24T16:49:50.025 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:49 smithi070 ceph-mon[81602]: pgmap v170: 161 pgs: 17 active+undersized+degraded, 13 peering, 2 stale+active+clean, 29 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-04-24T16:49:50.025 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:49 smithi070 podman[111685]: 2024-04-24 16:49:49.675430485 +0000 UTC m=+0.018149825 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:49:50.025 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:49 smithi070 podman[111685]: 2024-04-24 16:49:49.797543851 +0000 UTC m=+0.140263190 container create 9b691a28ab05ad98bbdc5293b2983ef0cac4d072501ba34b55ac76406de4f957 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-24T16:49:50.025 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 podman[111685]: 2024-04-24 16:49:50.025345778 +0000 UTC m=+0.368065121 container init 9b691a28ab05ad98bbdc5293b2983ef0cac4d072501ba34b55ac76406de4f957 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-24T16:49:50.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:49 smithi156 ceph-mon[65097]: pgmap v170: 161 pgs: 17 active+undersized+degraded, 13 peering, 2 stale+active+clean, 29 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 59/627 objects degraded (9.410%) 2024-04-24T16:49:50.293 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 podman[111685]: 2024-04-24 16:49:50.028455856 +0000 UTC m=+0.371175191 container start 9b691a28ab05ad98bbdc5293b2983ef0cac4d072501ba34b55ac76406de4f957 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-24T16:49:50.293 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 podman[111685]: 2024-04-24 16:49:50.142175392 +0000 UTC m=+0.484894733 container attach 9b691a28ab05ad98bbdc5293b2983ef0cac4d072501ba34b55ac76406de4f957 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T16:49:50.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:49:50.794 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:49:50.794 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:49:50.794 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:49:50.794 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T16:49:50.794 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-24T16:49:50.795 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-24T16:49:50.795 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-24T16:49:50.795 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-24T16:49:50.795 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T16:49:50.795 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 bash[111685]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-24T16:49:50.795 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T16:49:50.796 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-24T16:49:50.796 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-24T16:49:50.796 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-24T16:49:50.796 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-24T16:49:50.796 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T16:49:50.797 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate[111701]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-24T16:49:50.797 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:50 smithi070 podman[111685]: 2024-04-24 16:49:50.530869329 +0000 UTC m=+0.873588667 container died 9b691a28ab05ad98bbdc5293b2983ef0cac4d072501ba34b55ac76406de4f957 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True) 2024-04-24T16:49:51.112 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-mon[86496]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-24T16:49:51.112 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:51.112 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-mon[81602]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-24T16:49:51.112 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:50 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:51.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:50 smithi156 ceph-mon[65097]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-24T16:49:51.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:50 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:51.414 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 podman[111841]: 2024-04-24 16:49:51.112687534 +0000 UTC m=+0.576735382 container remove 9b691a28ab05ad98bbdc5293b2983ef0cac4d072501ba34b55ac76406de4f957 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-24T16:49:51.414 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 podman[111907]: 2024-04-24 16:49:51.313988741 +0000 UTC m=+0.102250776 container create 1c5293f7dc5747fbd7611883c01fac3dfc1cd66cd76e71c4963d05bee8a5b5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:49:51.414 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 podman[111907]: 2024-04-24 16:49:51.23029779 +0000 UTC m=+0.018559793 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:49:51.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 podman[111907]: 2024-04-24 16:49:51.5252546 +0000 UTC m=+0.313516596 container init 1c5293f7dc5747fbd7611883c01fac3dfc1cd66cd76e71c4963d05bee8a5b5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T16:49:51.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 podman[111907]: 2024-04-24 16:49:51.528369946 +0000 UTC m=+0.316631948 container start 1c5293f7dc5747fbd7611883c01fac3dfc1cd66cd76e71c4963d05bee8a5b5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-04-24T16:49:51.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-osd[111926]: -- 172.21.15.70:0/2019737787 <== mon.1 v2:172.21.15.70:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55650a381680 con 0x55650a34b000 2024-04-24T16:49:51.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 bash[111907]: 1c5293f7dc5747fbd7611883c01fac3dfc1cd66cd76e71c4963d05bee8a5b5fb 2024-04-24T16:49:51.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:51 smithi070 systemd[1]: Started Ceph osd.3 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:49:52.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:51 smithi156 ceph-mon[65097]: pgmap v171: 161 pgs: 18 active+undersized+degraded, 13 peering, 35 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-04-24T16:49:52.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:51 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:52.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:51 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:52.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:51 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:52.293 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:52 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:49:52.069+0000 7f06bab307c0 -1 Falling back to public interface 2024-04-24T16:49:52.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[81602]: pgmap v171: 161 pgs: 18 active+undersized+degraded, 13 peering, 35 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-04-24T16:49:52.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:52.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:52.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:52.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[86496]: pgmap v171: 161 pgs: 18 active+undersized+degraded, 13 peering, 35 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-04-24T16:49:52.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:52.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:52.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:51 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:49:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:53 smithi156 ceph-mon[65097]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-04-24T16:49:53.456 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:53 smithi070 ceph-mon[86496]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-04-24T16:49:53.457 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:53 smithi070 ceph-mon[81602]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-04-24T16:49:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:54 smithi156 ceph-mon[65097]: pgmap v172: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-24T16:49:54.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:54 smithi070 ceph-mon[86496]: pgmap v172: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-24T16:49:54.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:54 smithi070 ceph-mon[81602]: pgmap v172: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-24T16:49:55.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:49:54.789+0000 7f06bab307c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-24T16:49:55.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:49:54.815+0000 7f06bab307c0 -1 osd.3 127 log_to_monitors true 2024-04-24T16:49:55.393 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:55 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:55.394 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:55 smithi070 ceph-mon[86496]: from='osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:49:55.394 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:55 smithi070 ceph-mon[86496]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:49:55.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:55 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:55.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:55 smithi070 ceph-mon[81602]: from='osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:49:55.394 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:55 smithi070 ceph-mon[81602]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:49:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:55 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:49:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:55 smithi156 ceph-mon[65097]: from='osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:49:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:55 smithi156 ceph-mon[65097]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T16:49:55.793 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:49:55 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:49:55.393+0000 7f06a7f80700 -1 osd.3 127 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:49:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:56 smithi156 ceph-mon[65097]: pgmap v173: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-24T16:49:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:56 smithi156 ceph-mon[65097]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T16:49:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:56 smithi156 ceph-mon[65097]: osdmap e130: 8 total, 7 up, 8 in 2024-04-24T16:49:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:56 smithi156 ceph-mon[65097]: from='osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:56 smithi156 ceph-mon[65097]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:56.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:56 smithi156 ceph-mon[65097]: from='osd.3 ' entity='osd.3' 2024-04-24T16:49:56.519 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[86496]: pgmap v173: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[86496]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[86496]: osdmap e130: 8 total, 7 up, 8 in 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[86496]: from='osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[86496]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[86496]: from='osd.3 ' entity='osd.3' 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[81602]: pgmap v173: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[81602]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[81602]: osdmap e130: 8 total, 7 up, 8 in 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[81602]: from='osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[81602]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-24T16:49:56.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-mon[81602]: from='osd.3 ' entity='osd.3' 2024-04-24T16:49:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:49:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:49:56] "GET /metrics HTTP/1.1" 200 38250 "" "Prometheus/2.43.0" 2024-04-24T16:49:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:49:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474] boot 2024-04-24T16:49:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: osdmap e131: 8 total, 8 up, 8 in 2024-04-24T16:49:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:49:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: pgmap v176: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 24 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474] boot 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: osdmap e131: 8 total, 8 up, 8 in 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: pgmap v176: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-04-24T16:49:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:57 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 24 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:49:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: osd.3 [v2:172.21.15.70:6826/3831986474,v1:172.21.15.70:6827/3831986474] boot 2024-04-24T16:49:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: osdmap e131: 8 total, 8 up, 8 in 2024-04-24T16:49:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T16:49:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: pgmap v176: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-04-24T16:49:57.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:57 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 24 pgs degraded (PG_DEGRADED) 2024-04-24T16:49:58.510 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:58.510 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:58.510 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:58 smithi070 ceph-mon[86496]: osdmap e132: 8 total, 8 up, 8 in 2024-04-24T16:49:58.510 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:58.510 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:58.510 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:58 smithi070 ceph-mon[81602]: osdmap e132: 8 total, 8 up, 8 in 2024-04-24T16:49:58.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:58.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:49:58.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:58 smithi156 ceph-mon[65097]: osdmap e132: 8 total, 8 up, 8 in 2024-04-24T16:49:59.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:49:59 smithi070 ceph-mon[86496]: pgmap v178: 161 pgs: 8 peering, 10 active+undersized+degraded, 22 active+undersized, 121 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-04-24T16:49:59.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:49:59 smithi070 ceph-mon[81602]: pgmap v178: 161 pgs: 8 peering, 10 active+undersized+degraded, 22 active+undersized, 121 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-04-24T16:49:59.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:49:59 smithi156 ceph-mon[65097]: pgmap v178: 161 pgs: 8 peering, 10 active+undersized+degraded, 22 active+undersized, 121 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-04-24T16:50:00.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state']; Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded 2024-04-24T16:50:00.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:50:00.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:50:00.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: [WRN] PG_DEGRADED: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded 2024-04-24T16:50:00.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 3.b is active+undersized+degraded, acting [0,4] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 4.14 is active+undersized+degraded, acting [2,7] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 5.17 is active+undersized+degraded, acting [1,7] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[81602]: pg 6.c is active+undersized+degraded, acting [6,5] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state']; Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:50:00.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: [WRN] PG_DEGRADED: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 3.b is active+undersized+degraded, acting [0,4] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 4.14 is active+undersized+degraded, acting [2,7] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 5.17 is active+undersized+degraded, acting [1,7] 2024-04-24T16:50:00.545 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:00 smithi070 ceph-mon[86496]: pg 6.c is active+undersized+degraded, acting [6,5] 2024-04-24T16:50:00.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state']; Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded 2024-04-24T16:50:00.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:50:00.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:50:00.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: [WRN] PG_DEGRADED: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded 2024-04-24T16:50:00.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 3.b is active+undersized+degraded, acting [0,4] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 4.14 is active+undersized+degraded, acting [2,7] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 5.17 is active+undersized+degraded, acting [1,7] 2024-04-24T16:50:00.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:00 smithi156 ceph-mon[65097]: pg 6.c is active+undersized+degraded, acting [6,5] 2024-04-24T16:50:01.293 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:01 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:50:01.239+0000 7f22938a6700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-24T16:50:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: pgmap v179: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: pgmap v180: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: pgmap v181: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:02.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: pgmap v179: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: pgmap v180: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: pgmap v181: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: pgmap v179: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: pgmap v180: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: pgmap v181: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:02.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:02.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:02.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-04-24T16:50:02.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:02.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:03 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:03 smithi156 ceph-mon[65097]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-24T16:50:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:03 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:03 smithi070 ceph-mon[86496]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-24T16:50:03.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:03 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:03.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:03 smithi070 ceph-mon[81602]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-04-24T16:50:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:04 smithi156 ceph-mon[65097]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:04.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:04 smithi070 ceph-mon[86496]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:04.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:04 smithi070 ceph-mon[81602]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:05 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:05.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:05 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:05 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:06.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:06 smithi156 ceph-mon[65097]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:06.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:06 smithi070 ceph-mon[86496]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:06.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:06 smithi070 ceph-mon[81602]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:50:06] "GET /metrics HTTP/1.1" 200 38260 "" "Prometheus/2.43.0" 2024-04-24T16:50:08.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:08 smithi156 ceph-mon[65097]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-24T16:50:08.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:08 smithi070 ceph-mon[86496]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-24T16:50:08.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:08 smithi070 ceph-mon[81602]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 253 B/s rd, 0 op/s 2024-04-24T16:50:10.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:10 smithi156 ceph-mon[65097]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-04-24T16:50:10.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:10 smithi070 ceph-mon[86496]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-04-24T16:50:10.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:10 smithi070 ceph-mon[81602]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-04-24T16:50:12.362 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:12 smithi070 ceph-mon[86496]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:50:12.362 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:12 smithi070 ceph-mon[81602]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:50:12.363 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:50:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:12 smithi156 ceph-mon[65097]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (6m) 16s ago 28m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (6m) 4m ago 28m 79.9M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 16s ago 28m 64.0M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (8m) 4m ago 33m 449M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (18m) 16s ago 34m 509M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (4m) 16s ago 34m 51.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (5m) 4m ago 33m 38.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:50:13.014 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (4m) 16s ago 33m 44.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (7m) 16s ago 29m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (7m) 4m ago 29m 15.9M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (2m) 16s ago 32m 52.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (2m) 16s ago 32m 54.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (77s) 16s ago 32m 44.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (21s) 16s ago 31m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (31m) 4m ago 31m 54.9M 3248M 17.2.0 e1d6a67b021e ed3cf700bc24 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (31m) 4m ago 31m 53.5M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (30m) 4m ago 30m 50.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (30m) 4m ago 30m 55.4M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (7m) 4m ago 29m 66.7M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (28m) 16s ago 28m 78.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:50:13.015 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (28m) 4m ago 28m 76.4M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:50:13.413 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 4 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:13.414 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:50:13.415 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-24T16:50:13.415 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 9 2024-04-24T16:50:13.415 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:50:13.415 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:50:13.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:13 smithi156 ceph-mon[65097]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:13 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:13.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:13 smithi156 ceph-mon[65097]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:13 smithi156 ceph-mon[65097]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:13 smithi156 ceph-mon[65097]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "4/8 daemons upgraded", 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:50:13.749 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:50:13.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[86496]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:13.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[86496]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[86496]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.750 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[86496]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:13.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[81602]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:13.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[81602]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[81602]: from='client.54483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:13.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:13 smithi070 ceph-mon[81602]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:14.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:14 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:14.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:14 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1564906843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:14.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:14 smithi156 ceph-mon[65097]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:14.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:14 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:14.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:14 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1564906843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:14.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:14 smithi070 ceph-mon[81602]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:14.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:14 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:14.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:14 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1564906843' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:14.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:14 smithi070 ceph-mon[86496]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:15 smithi070 ceph-mon[81602]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:15.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:15 smithi070 ceph-mon[86496]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:15.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:15 smithi156 ceph-mon[65097]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:16.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:16.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:16.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:50:16] "GET /metrics HTTP/1.1" 200 38260 "" "Prometheus/2.43.0" 2024-04-24T16:50:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:17 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:17 smithi156 ceph-mon[65097]: Upgrade: osd.4 is safe to restart 2024-04-24T16:50:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:17 smithi156 ceph-mon[65097]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:17.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:17.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[86496]: Upgrade: osd.4 is safe to restart 2024-04-24T16:50:17.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:17.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[86496]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:17.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-24T16:50:17.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[81602]: Upgrade: osd.4 is safe to restart 2024-04-24T16:50:17.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:17.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:17 smithi070 ceph-mon[81602]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:19.134 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:18 smithi156 ceph-mon[65097]: Upgrade: Updating osd.4 2024-04-24T16:50:19.135 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:19.135 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:50:19.135 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:19.135 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:18 smithi156 ceph-mon[65097]: Deploying daemon osd.4 on smithi156 2024-04-24T16:50:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[86496]: Upgrade: Updating osd.4 2024-04-24T16:50:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:50:19.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:19.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[86496]: Deploying daemon osd.4 on smithi156 2024-04-24T16:50:19.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[81602]: Upgrade: Updating osd.4 2024-04-24T16:50:19.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:19.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T16:50:19.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:19.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:18 smithi070 ceph-mon[81602]: Deploying daemon osd.4 on smithi156 2024-04-24T16:50:20.110 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:19 smithi156 ceph-mon[65097]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:20.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:19 smithi070 ceph-mon[86496]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:20.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:19 smithi070 ceph-mon[81602]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:21.397 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:21 smithi156 systemd[1]: Stopping Ceph osd.4 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:50:21.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:21 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[26116]: 2024-04-24T16:50:21.396+0000 7f8ace68c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:50:21.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:21 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[26116]: 2024-04-24T16:50:21.396+0000 7f8ace68c700 -1 osd.4 132 *** Got signal Terminated *** 2024-04-24T16:50:21.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:21 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[26116]: 2024-04-24T16:50:21.396+0000 7f8ace68c700 -1 osd.4 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:50:22.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:22 smithi070 ceph-mon[86496]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:22.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:22 smithi070 ceph-mon[86496]: osd.4 marked itself down and dead 2024-04-24T16:50:22.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:22 smithi070 ceph-mon[81602]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:22.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:22 smithi070 ceph-mon[81602]: osd.4 marked itself down and dead 2024-04-24T16:50:22.562 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:22 smithi156 ceph-mon[65097]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:22.562 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:22 smithi156 ceph-mon[65097]: osd.4 marked itself down and dead 2024-04-24T16:50:22.910 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:22 smithi156 podman[75439]: 2024-04-24 16:50:22.708898013 +0000 UTC m=+1.420179344 container died ed3cf700bc24333c9ba95624afc0a1593e505a8cf2ffb6d768efe0cb2110ebaf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, release=754, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-24T16:50:23.364 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:23 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:50:23.364 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:23 smithi156 ceph-mon[65097]: osdmap e133: 8 total, 7 up, 8 in 2024-04-24T16:50:23.364 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:23 smithi156 ceph-mon[65097]: osdmap e134: 8 total, 7 up, 8 in 2024-04-24T16:50:23.364 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 podman[75439]: 2024-04-24 16:50:23.096592583 +0000 UTC m=+1.807873924 container cleanup ed3cf700bc24333c9ba95624afc0a1593e505a8cf2ffb6d768efe0cb2110ebaf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-24T16:50:23.364 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 bash[75439]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4 2024-04-24T16:50:23.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:23 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:50:23.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:23 smithi070 ceph-mon[86496]: osdmap e133: 8 total, 7 up, 8 in 2024-04-24T16:50:23.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:23 smithi070 ceph-mon[86496]: osdmap e134: 8 total, 7 up, 8 in 2024-04-24T16:50:23.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:23 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:50:23.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:23 smithi070 ceph-mon[81602]: osdmap e133: 8 total, 7 up, 8 in 2024-04-24T16:50:23.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:23 smithi070 ceph-mon[81602]: osdmap e134: 8 total, 7 up, 8 in 2024-04-24T16:50:23.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 podman[75458]: 2024-04-24 16:50:23.364766358 +0000 UTC m=+0.654919722 container remove ed3cf700bc24333c9ba95624afc0a1593e505a8cf2ffb6d768efe0cb2110ebaf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, ceph=True, io.k8s.display-name=CentOS Stream 8) 2024-04-24T16:50:23.934 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 podman[75548]: 2024-04-24 16:50:23.571424824 +0000 UTC m=+0.018167866 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:50:23.934 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 podman[75548]: 2024-04-24 16:50:23.682549735 +0000 UTC m=+0.129292819 container create 09187c45954e0ba1fb45ac12c738a3c0917a5651e1a68644d1c8798546d0e9c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:50:23.934 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 podman[75548]: 2024-04-24 16:50:23.876986716 +0000 UTC m=+0.323729763 container init 09187c45954e0ba1fb45ac12c738a3c0917a5651e1a68644d1c8798546d0e9c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-24T16:50:23.934 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 podman[75548]: 2024-04-24 16:50:23.880712971 +0000 UTC m=+0.327456023 container start 09187c45954e0ba1fb45ac12c738a3c0917a5651e1a68644d1c8798546d0e9c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:50:24.411 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:23 smithi156 podman[75548]: 2024-04-24 16:50:23.934818346 +0000 UTC m=+0.381561411 container attach 09187c45954e0ba1fb45ac12c738a3c0917a5651e1a68644d1c8798546d0e9c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T16:50:24.411 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:24 smithi156 podman[75548]: 2024-04-24 16:50:24.174293875 +0000 UTC m=+0.621036919 container died 09187c45954e0ba1fb45ac12c738a3c0917a5651e1a68644d1c8798546d0e9c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-04-24T16:50:24.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:24 smithi156 ceph-mon[65097]: pgmap v193: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-24T16:50:24.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:24 smithi156 ceph-mon[65097]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-04-24T16:50:24.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:24 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:24.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:24 smithi070 ceph-mon[86496]: pgmap v193: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-24T16:50:24.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:24 smithi070 ceph-mon[86496]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-04-24T16:50:24.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:24 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:24.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:24 smithi070 ceph-mon[81602]: pgmap v193: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-24T16:50:24.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:24 smithi070 ceph-mon[81602]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-04-24T16:50:24.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:24 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:25.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:24 smithi156 podman[75575]: 2024-04-24 16:50:24.730225151 +0000 UTC m=+0.550342332 container remove 09187c45954e0ba1fb45ac12c738a3c0917a5651e1a68644d1c8798546d0e9c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T16:50:25.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:24 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-24T16:50:25.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:24 smithi156 systemd[1]: Stopped Ceph osd.4 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:50:25.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:24 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.4.service: Consumed 17.175s CPU time. 2024-04-24T16:50:25.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:24 smithi156 systemd[1]: Starting Ceph osd.4 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:50:25.421 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-mon[65097]: pgmap v195: 161 pgs: 41 peering, 12 stale+active+clean, 108 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:25.421 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 podman[75679]: 2024-04-24 16:50:25.073754326 +0000 UTC m=+0.018184880 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:50:25.421 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 podman[75679]: 2024-04-24 16:50:25.199764842 +0000 UTC m=+0.144195364 container create d34e6ffa7683be985e5b7354eee7d0aec4d83d1ffaac927cb421a92822068da4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-24T16:50:25.422 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 podman[75679]: 2024-04-24 16:50:25.417787045 +0000 UTC m=+0.362217572 container init d34e6ffa7683be985e5b7354eee7d0aec4d83d1ffaac927cb421a92822068da4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:50:25.422 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 podman[75679]: 2024-04-24 16:50:25.421991805 +0000 UTC m=+0.366422328 container start d34e6ffa7683be985e5b7354eee7d0aec4d83d1ffaac927cb421a92822068da4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:50:25.439 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:25 smithi070 ceph-mon[86496]: pgmap v195: 161 pgs: 41 peering, 12 stale+active+clean, 108 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:25.439 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:25 smithi070 ceph-mon[81602]: pgmap v195: 161 pgs: 41 peering, 12 stale+active+clean, 108 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:25.903 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 podman[75679]: 2024-04-24 16:50:25.501174742 +0000 UTC m=+0.445605274 container attach d34e6ffa7683be985e5b7354eee7d0aec4d83d1ffaac927cb421a92822068da4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-24T16:50:26.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 bash[75679]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate[75693]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-24T16:50:26.162 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:25 smithi156 podman[75679]: 2024-04-24 16:50:25.928599344 +0000 UTC m=+0.873029878 container died d34e6ffa7683be985e5b7354eee7d0aec4d83d1ffaac927cb421a92822068da4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-24T16:50:26.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:50:26] "GET /metrics HTTP/1.1" 200 38263 "" "Prometheus/2.43.0" 2024-04-24T16:50:26.856 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:26 smithi156 podman[75820]: 2024-04-24 16:50:26.505275774 +0000 UTC m=+0.568390977 container remove d34e6ffa7683be985e5b7354eee7d0aec4d83d1ffaac927cb421a92822068da4 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:50:26.856 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:26 smithi156 podman[75885]: 2024-04-24 16:50:26.624276848 +0000 UTC m=+0.018372963 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:50:26.856 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:26 smithi156 podman[75885]: 2024-04-24 16:50:26.740273045 +0000 UTC m=+0.134369206 container create 8b08bde60952e0c95de10c9a86ed8ca4b291f0011b8bbc8226c56a53501d6b0a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:50:27.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:26 smithi156 podman[75885]: 2024-04-24 16:50:26.975966146 +0000 UTC m=+0.370062266 container init 8b08bde60952e0c95de10c9a86ed8ca4b291f0011b8bbc8226c56a53501d6b0a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-24T16:50:27.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:26 smithi156 podman[75885]: 2024-04-24 16:50:26.97911429 +0000 UTC m=+0.373210398 container start 8b08bde60952e0c95de10c9a86ed8ca4b291f0011b8bbc8226c56a53501d6b0a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:50:27.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:26 smithi156 ceph-osd[75904]: -- 172.21.15.156:0/2528102385 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564337cf9680 con 0x564337cc3000 2024-04-24T16:50:27.160 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:27 smithi156 bash[75885]: 8b08bde60952e0c95de10c9a86ed8ca4b291f0011b8bbc8226c56a53501d6b0a 2024-04-24T16:50:27.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:27 smithi156 systemd[1]: Started Ceph osd.4 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:50:27.910 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:50:27.521+0000 7f85d81a87c0 -1 Falling back to public interface 2024-04-24T16:50:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:28 smithi156 ceph-mon[65097]: pgmap v196: 161 pgs: 4 active+undersized+degraded, 41 peering, 3 stale+active+clean, 13 active+undersized, 100 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 11/627 objects degraded (1.754%) 2024-04-24T16:50:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:28 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 11/627 objects degraded (1.754%), 4 pgs degraded (PG_DEGRADED) 2024-04-24T16:50:28.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:28.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[81602]: pgmap v196: 161 pgs: 4 active+undersized+degraded, 41 peering, 3 stale+active+clean, 13 active+undersized, 100 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 11/627 objects degraded (1.754%) 2024-04-24T16:50:28.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:28.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:28.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 11/627 objects degraded (1.754%), 4 pgs degraded (PG_DEGRADED) 2024-04-24T16:50:28.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:28.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[86496]: pgmap v196: 161 pgs: 4 active+undersized+degraded, 41 peering, 3 stale+active+clean, 13 active+undersized, 100 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 11/627 objects degraded (1.754%) 2024-04-24T16:50:28.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:28.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:28.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 11/627 objects degraded (1.754%), 4 pgs degraded (PG_DEGRADED) 2024-04-24T16:50:28.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:29.525 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:29 smithi156 ceph-mon[65097]: pgmap v197: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-24T16:50:29.525 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:29 smithi156 ceph-mon[65097]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 11 pgs peering) 2024-04-24T16:50:29.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:29 smithi070 ceph-mon[86496]: pgmap v197: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-24T16:50:29.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:29 smithi070 ceph-mon[86496]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 11 pgs peering) 2024-04-24T16:50:29.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:29 smithi070 ceph-mon[81602]: pgmap v197: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-24T16:50:29.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:29 smithi070 ceph-mon[81602]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 11 pgs peering) 2024-04-24T16:50:30.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:30 smithi156 ceph-mon[65097]: from='osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:50:30.622 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:30 smithi156 ceph-mon[65097]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:50:30.622 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:50:30.252+0000 7f85d81a87c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-24T16:50:30.623 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:50:30.281+0000 7f85d81a87c0 -1 osd.4 132 log_to_monitors true 2024-04-24T16:50:30.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:30 smithi070 ceph-mon[86496]: from='osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:50:30.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:30 smithi070 ceph-mon[86496]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:50:30.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:30 smithi070 ceph-mon[81602]: from='osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:50:30.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:30 smithi070 ceph-mon[81602]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T16:50:31.345 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:50:30 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:50:30.622+0000 7f85c55f8700 -1 osd.4 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:50:31.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:31 smithi156 ceph-mon[65097]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T16:50:31.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:31 smithi156 ceph-mon[65097]: from='osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:50:31.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:31 smithi156 ceph-mon[65097]: osdmap e135: 8 total, 7 up, 8 in 2024-04-24T16:50:31.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:31 smithi156 ceph-mon[65097]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:50:31.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:31 smithi156 ceph-mon[65097]: from='osd.4 ' entity='osd.4' 2024-04-24T16:50:31.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:31 smithi156 ceph-mon[65097]: pgmap v199: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-24T16:50:31.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[86496]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T16:50:31.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[86496]: from='osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:50:31.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[86496]: osdmap e135: 8 total, 7 up, 8 in 2024-04-24T16:50:31.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[86496]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:50:31.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[86496]: from='osd.4 ' entity='osd.4' 2024-04-24T16:50:31.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[86496]: pgmap v199: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-24T16:50:31.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[81602]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T16:50:31.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[81602]: from='osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:50:31.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[81602]: osdmap e135: 8 total, 7 up, 8 in 2024-04-24T16:50:31.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[81602]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:50:31.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[81602]: from='osd.4 ' entity='osd.4' 2024-04-24T16:50:31.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:31 smithi070 ceph-mon[81602]: pgmap v199: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-24T16:50:32.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:50:32.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291] boot 2024-04-24T16:50:32.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: osdmap e136: 8 total, 8 up, 8 in 2024-04-24T16:50:32.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:50:32.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291] boot 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: osdmap e136: 8 total, 8 up, 8 in 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:32.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:50:32.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: osd.4 [v2:172.21.15.156:6800/4065489291,v1:172.21.15.156:6801/4065489291] boot 2024-04-24T16:50:32.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: osdmap e136: 8 total, 8 up, 8 in 2024-04-24T16:50:32.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T16:50:32.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:32.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:33.832 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:33.833 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:33.833 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:33 smithi156 ceph-mon[65097]: osdmap e137: 8 total, 8 up, 8 in 2024-04-24T16:50:33.833 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:33 smithi156 ceph-mon[65097]: pgmap v202: 161 pgs: 7 peering, 14 active+undersized+degraded, 24 active+undersized, 116 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 47/627 objects degraded (7.496%) 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[81602]: osdmap e137: 8 total, 8 up, 8 in 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[81602]: pgmap v202: 161 pgs: 7 peering, 14 active+undersized+degraded, 24 active+undersized, 116 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 47/627 objects degraded (7.496%) 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[86496]: osdmap e137: 8 total, 8 up, 8 in 2024-04-24T16:50:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:33 smithi070 ceph-mon[86496]: pgmap v202: 161 pgs: 7 peering, 14 active+undersized+degraded, 24 active+undersized, 116 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 47/627 objects degraded (7.496%) 2024-04-24T16:50:34.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:34 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:34.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:34 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 47/627 objects degraded (7.496%), 14 pgs degraded (PG_DEGRADED) 2024-04-24T16:50:35.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:34 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:35.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:34 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 47/627 objects degraded (7.496%), 14 pgs degraded (PG_DEGRADED) 2024-04-24T16:50:35.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:34 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:35.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:34 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 47/627 objects degraded (7.496%), 14 pgs degraded (PG_DEGRADED) 2024-04-24T16:50:35.635 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:35 smithi156 ceph-mon[65097]: pgmap v203: 161 pgs: 7 peering, 11 active+undersized+degraded, 20 active+undersized, 123 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 32/627 objects degraded (5.104%) 2024-04-24T16:50:36.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:35 smithi070 ceph-mon[86496]: pgmap v203: 161 pgs: 7 peering, 11 active+undersized+degraded, 20 active+undersized, 123 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 32/627 objects degraded (5.104%) 2024-04-24T16:50:36.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:35 smithi070 ceph-mon[81602]: pgmap v203: 161 pgs: 7 peering, 11 active+undersized+degraded, 20 active+undersized, 123 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 32/627 objects degraded (5.104%) 2024-04-24T16:50:36.520 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:50:36.193+0000 7f22938a6700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-24T16:50:36.520 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:50:36] "GET /metrics HTTP/1.1" 200 38132 "" "Prometheus/2.43.0" 2024-04-24T16:50:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:50:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: pgmap v204: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-04-24T16:50:37.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: pgmap v205: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: pgmap v204: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: pgmap v205: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:50:37.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:37.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:50:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: pgmap v204: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-04-24T16:50:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: pgmap v205: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:50:37.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:37.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:50:37.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:37.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:38.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:38 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:38.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:38 smithi070 ceph-mon[81602]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-24T16:50:38.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:38 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/627 objects degraded (5.104%), 11 pgs degraded) 2024-04-24T16:50:38.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:38 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:38.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:38 smithi070 ceph-mon[86496]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-24T16:50:38.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:38 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/627 objects degraded (5.104%), 11 pgs degraded) 2024-04-24T16:50:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:38 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:38 smithi156 ceph-mon[65097]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-04-24T16:50:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:38 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/627 objects degraded (5.104%), 11 pgs degraded) 2024-04-24T16:50:39.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:39 smithi070 ceph-mon[81602]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-04-24T16:50:39.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:39 smithi070 ceph-mon[86496]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-04-24T16:50:39.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:39 smithi156 ceph-mon[65097]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 0 op/s 2024-04-24T16:50:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:41 smithi156 ceph-mon[65097]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-04-24T16:50:41.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:41 smithi070 ceph-mon[86496]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-04-24T16:50:41.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:41 smithi070 ceph-mon[81602]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-04-24T16:50:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:43 smithi156 ceph-mon[65097]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:50:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:43 smithi070 ceph-mon[86496]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:50:43.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:43 smithi070 ceph-mon[81602]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:50:44.112 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:50:44.758 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:50:44.758 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (7m) 48s ago 29m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:50:44.758 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (6m) 13s ago 29m 81.0M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:50:44.758 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 48s ago 28m 64.0M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:50:44.758 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (8m) 13s ago 33m 453M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (18m) 48s ago 35m 509M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (5m) 48s ago 35m 51.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (6m) 13s ago 34m 46.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (4m) 48s ago 34m 44.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (8m) 48s ago 30m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (8m) 13s ago 30m 15.4M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (3m) 48s ago 33m 52.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (2m) 48s ago 33m 54.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (109s) 48s ago 32m 44.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:50:44.759 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (53s) 48s ago 32m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:50:44.760 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (18s) 13s ago 31m 11.8M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:50:44.760 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (31m) 13s ago 31m 57.4M 3248M 17.2.0 e1d6a67b021e 74908fdd6f6e 2024-04-24T16:50:44.760 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (31m) 13s ago 31m 53.8M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:50:44.760 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (30m) 13s ago 30m 60.1M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:50:44.760 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (7m) 13s ago 30m 69.3M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:50:44.760 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (28m) 48s ago 28m 78.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:50:44.760 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (28m) 13s ago 28m 78.0M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:50:45.159 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 5 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:50:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:50:45.161 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:50:45.161 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-24T16:50:45.161 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 10 2024-04-24T16:50:45.161 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:50:45.161 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:50:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:45 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:45.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:45 smithi156 ceph-mon[65097]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:50:45.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:45 smithi156 ceph-mon[65097]: from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:45.498 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:50:45.498 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:50:45.498 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:50:45.498 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-24T16:50:45.498 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:50:45.498 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "5/8 daemons upgraded", 2024-04-24T16:50:45.498 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-04-24T16:50:45.499 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:50:45.499 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:50:45.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:45 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:45.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:45 smithi070 ceph-mon[81602]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:50:45.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:45 smithi070 ceph-mon[81602]: from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:45.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:45 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:45.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:45 smithi070 ceph-mon[86496]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:50:45.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:45 smithi070 ceph-mon[86496]: from='client.64441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:46 smithi156 ceph-mon[65097]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:46 smithi156 ceph-mon[65097]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:46 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2843032571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:46.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:46 smithi070 ceph-mon[86496]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:46.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:46 smithi070 ceph-mon[86496]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:46.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:46 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2843032571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:46.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:46 smithi070 ceph-mon[81602]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:46.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:46 smithi070 ceph-mon[81602]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:46.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:46 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2843032571' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:50:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:50:46] "GET /metrics HTTP/1.1" 200 38132 "" "Prometheus/2.43.0" 2024-04-24T16:50:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:47 smithi156 ceph-mon[65097]: from='client.54528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:47 smithi156 ceph-mon[65097]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:50:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[81602]: from='client.54528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[81602]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:50:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[86496]: from='client.54528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:50:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[86496]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:50:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:50:48.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:48.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:48 smithi156 ceph-mon[65097]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:48.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:48.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:48 smithi070 ceph-mon[86496]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:48.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:48.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:48 smithi070 ceph-mon[81602]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:51 smithi156 ceph-mon[65097]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:51.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:51 smithi070 ceph-mon[86496]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:51 smithi070 ceph-mon[81602]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:52 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:52.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:52 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:52.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:52 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:53 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:53 smithi156 ceph-mon[65097]: Upgrade: osd.5 is safe to restart 2024-04-24T16:50:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:53 smithi156 ceph-mon[65097]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:53 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:53 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:50:53.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:53 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[86496]: Upgrade: osd.5 is safe to restart 2024-04-24T16:50:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[86496]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:50:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:53.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-24T16:50:53.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[81602]: Upgrade: osd.5 is safe to restart 2024-04-24T16:50:53.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[81602]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:53.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:50:53.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T16:50:53.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:53 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:50:54.274 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:54 smithi156 ceph-mon[65097]: Upgrade: Updating osd.5 2024-04-24T16:50:54.274 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:54 smithi156 ceph-mon[65097]: Deploying daemon osd.5 on smithi156 2024-04-24T16:50:54.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:54 smithi070 ceph-mon[86496]: Upgrade: Updating osd.5 2024-04-24T16:50:54.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:54 smithi070 ceph-mon[86496]: Deploying daemon osd.5 on smithi156 2024-04-24T16:50:54.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:54 smithi070 ceph-mon[81602]: Upgrade: Updating osd.5 2024-04-24T16:50:54.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:54 smithi070 ceph-mon[81602]: Deploying daemon osd.5 on smithi156 2024-04-24T16:50:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:55 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:55 smithi156 ceph-mon[65097]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:55 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:55 smithi070 ceph-mon[81602]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:55.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:55 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:50:55.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:55 smithi070 ceph-mon[86496]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:50:56.512 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:56 smithi156 systemd[1]: Stopping Ceph osd.5 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:50:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:50:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:50:56] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.43.0" 2024-04-24T16:50:56.910 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[29004]: 2024-04-24T16:50:56.511+0000 7fa308b70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:50:56.910 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[29004]: 2024-04-24T16:50:56.511+0000 7fa308b70700 -1 osd.5 137 *** Got signal Terminated *** 2024-04-24T16:50:56.910 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[29004]: 2024-04-24T16:50:56.511+0000 7fa308b70700 -1 osd.5 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:50:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:57 smithi156 ceph-mon[65097]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:57 smithi156 ceph-mon[65097]: osd.5 marked itself down and dead 2024-04-24T16:50:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:57 smithi070 ceph-mon[86496]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:57 smithi070 ceph-mon[86496]: osd.5 marked itself down and dead 2024-04-24T16:50:57.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:57 smithi070 ceph-mon[81602]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:50:57.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:57 smithi070 ceph-mon[81602]: osd.5 marked itself down and dead 2024-04-24T16:50:57.787 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:57 smithi156 podman[80291]: 2024-04-24 16:50:57.502320426 +0000 UTC m=+1.096262478 container died 74908fdd6f6e1925df01adaac8449624c005bdb270d43ce7f54ea8b55ecb046f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, distribution-scope=public, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0) 2024-04-24T16:50:58.070 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:57 smithi156 podman[80291]: 2024-04-24 16:50:57.861610512 +0000 UTC m=+1.455552573 container cleanup 74908fdd6f6e1925df01adaac8449624c005bdb270d43ce7f54ea8b55ecb046f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, name=centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, release=754, vendor=Red Hat, Inc., ceph=True, com.redhat.component=centos-stream-container) 2024-04-24T16:50:58.071 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:57 smithi156 bash[80291]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5 2024-04-24T16:50:58.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:58 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:50:58.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:58 smithi156 ceph-mon[65097]: osdmap e138: 8 total, 7 up, 8 in 2024-04-24T16:50:58.411 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:58 smithi156 podman[80307]: 2024-04-24 16:50:58.071333864 +0000 UTC m=+0.568965198 container remove 74908fdd6f6e1925df01adaac8449624c005bdb270d43ce7f54ea8b55ecb046f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , release=754, vcs-type=git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-24T16:50:58.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:58 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:50:58.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:58 smithi070 ceph-mon[86496]: osdmap e138: 8 total, 7 up, 8 in 2024-04-24T16:50:58.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:58 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:50:58.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:58 smithi070 ceph-mon[81602]: osdmap e138: 8 total, 7 up, 8 in 2024-04-24T16:50:58.741 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:58 smithi156 podman[80400]: 2024-04-24 16:50:58.391336321 +0000 UTC m=+0.018848991 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:50:58.741 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:58 smithi156 podman[80400]: 2024-04-24 16:50:58.522494136 +0000 UTC m=+0.150006784 container create e78c2dcc85de5d1cf362d9c70af9a956edb4099b32966cbe84c63215b229be1a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T16:50:59.041 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:58 smithi156 podman[80400]: 2024-04-24 16:50:58.742004137 +0000 UTC m=+0.369516791 container init e78c2dcc85de5d1cf362d9c70af9a956edb4099b32966cbe84c63215b229be1a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-24T16:50:59.041 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:58 smithi156 podman[80400]: 2024-04-24 16:50:58.745674092 +0000 UTC m=+0.373186747 container start e78c2dcc85de5d1cf362d9c70af9a956edb4099b32966cbe84c63215b229be1a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:50:59.042 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:58 smithi156 podman[80400]: 2024-04-24 16:50:58.824842555 +0000 UTC m=+0.452355211 container attach e78c2dcc85de5d1cf362d9c70af9a956edb4099b32966cbe84c63215b229be1a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-24T16:50:59.378 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:59 smithi156 podman[80400]: 2024-04-24 16:50:59.04266659 +0000 UTC m=+0.670179256 container died e78c2dcc85de5d1cf362d9c70af9a956edb4099b32966cbe84c63215b229be1a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-24T16:50:59.378 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:59 smithi156 ceph-mon[65097]: pgmap v217: 161 pgs: 28 peering, 11 stale+active+clean, 122 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:59.378 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:59 smithi156 ceph-mon[65097]: Health check failed: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-04-24T16:50:59.378 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:50:59 smithi156 ceph-mon[65097]: osdmap e139: 8 total, 7 up, 8 in 2024-04-24T16:50:59.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:59 smithi070 ceph-mon[86496]: pgmap v217: 161 pgs: 28 peering, 11 stale+active+clean, 122 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:59.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:59 smithi070 ceph-mon[86496]: Health check failed: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-04-24T16:50:59.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:50:59 smithi070 ceph-mon[86496]: osdmap e139: 8 total, 7 up, 8 in 2024-04-24T16:50:59.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:59 smithi070 ceph-mon[81602]: pgmap v217: 161 pgs: 28 peering, 11 stale+active+clean, 122 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:50:59.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:59 smithi070 ceph-mon[81602]: Health check failed: Reduced data availability: 7 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-04-24T16:50:59.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:50:59 smithi070 ceph-mon[81602]: osdmap e139: 8 total, 7 up, 8 in 2024-04-24T16:50:59.910 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:59 smithi156 podman[80425]: 2024-04-24 16:50:59.637137865 +0000 UTC m=+0.585952984 container remove e78c2dcc85de5d1cf362d9c70af9a956edb4099b32966cbe84c63215b229be1a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-24T16:50:59.911 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:59 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-24T16:50:59.911 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:59 smithi156 systemd[1]: Stopped Ceph osd.5 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:50:59.911 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:59 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.5.service: Consumed 15.698s CPU time. 2024-04-24T16:50:59.911 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:50:59 smithi156 systemd[1]: Starting Ceph osd.5 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:51:00.332 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 podman[80529]: 2024-04-24 16:50:59.975518374 +0000 UTC m=+0.018418119 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:51:00.332 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 podman[80529]: 2024-04-24 16:51:00.088243813 +0000 UTC m=+0.131143559 container create 1420c06a8b8cb22b5468bc6447e1f20e16e08a11faa7f99e3b85fa20dd75e7ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T16:51:00.660 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 podman[80529]: 2024-04-24 16:51:00.332415821 +0000 UTC m=+0.375315580 container init 1420c06a8b8cb22b5468bc6447e1f20e16e08a11faa7f99e3b85fa20dd75e7ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-24T16:51:00.660 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 podman[80529]: 2024-04-24 16:51:00.336146121 +0000 UTC m=+0.379045869 container start 1420c06a8b8cb22b5468bc6447e1f20e16e08a11faa7f99e3b85fa20dd75e7ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-24T16:51:00.660 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 podman[80529]: 2024-04-24 16:51:00.415786408 +0000 UTC m=+0.458686163 container attach 1420c06a8b8cb22b5468bc6447e1f20e16e08a11faa7f99e3b85fa20dd75e7ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-24T16:51:01.126 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 bash[80529]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate[80544]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-24T16:51:01.127 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:00 smithi156 podman[80529]: 2024-04-24 16:51:00.839786938 +0000 UTC m=+0.882686684 container died 1420c06a8b8cb22b5468bc6447e1f20e16e08a11faa7f99e3b85fa20dd75e7ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:51:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:01 smithi156 ceph-mon[65097]: pgmap v219: 161 pgs: 34 peering, 10 stale+active+clean, 117 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:51:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:01 smithi070 ceph-mon[86496]: pgmap v219: 161 pgs: 34 peering, 10 stale+active+clean, 117 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:51:01.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:01 smithi070 ceph-mon[81602]: pgmap v219: 161 pgs: 34 peering, 10 stale+active+clean, 117 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:51:01.849 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:01 smithi156 podman[80667]: 2024-04-24 16:51:01.511964991 +0000 UTC m=+0.666184296 container remove 1420c06a8b8cb22b5468bc6447e1f20e16e08a11faa7f99e3b85fa20dd75e7ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:51:01.849 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:01 smithi156 podman[80735]: 2024-04-24 16:51:01.636154528 +0000 UTC m=+0.018548217 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:51:01.849 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:01 smithi156 podman[80735]: 2024-04-24 16:51:01.746173895 +0000 UTC m=+0.128567552 container create 4ab2caf74886615c99d860b08045cf951aebeecd62c178d70602bda0f170b787 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:51:02.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:01 smithi156 podman[80735]: 2024-04-24 16:51:01.95678287 +0000 UTC m=+0.339176527 container init 4ab2caf74886615c99d860b08045cf951aebeecd62c178d70602bda0f170b787 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:51:02.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:01 smithi156 podman[80735]: 2024-04-24 16:51:01.959931247 +0000 UTC m=+0.342324905 container start 4ab2caf74886615c99d860b08045cf951aebeecd62c178d70602bda0f170b787 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-24T16:51:02.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:01 smithi156 ceph-osd[80754]: -- 172.21.15.156:0/2566836006 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0cec29680 con 0x55b0cebf3000 2024-04-24T16:51:02.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:02 smithi156 bash[80735]: 4ab2caf74886615c99d860b08045cf951aebeecd62c178d70602bda0f170b787 2024-04-24T16:51:02.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:02 smithi156 systemd[1]: Started Ceph osd.5 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:51:02.902 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:51:02.509+0000 7f1c2706c7c0 -1 Falling back to public interface 2024-04-24T16:51:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:03 smithi156 ceph-mon[65097]: pgmap v220: 161 pgs: 10 active+undersized+degraded, 34 peering, 15 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 36/627 objects degraded (5.742%) 2024-04-24T16:51:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:03.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:03 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:03.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:03 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[86496]: pgmap v220: 161 pgs: 10 active+undersized+degraded, 34 peering, 15 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 36/627 objects degraded (5.742%) 2024-04-24T16:51:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[81602]: pgmap v220: 161 pgs: 10 active+undersized+degraded, 34 peering, 15 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 36/627 objects degraded (5.742%) 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 36/627 objects degraded (5.742%), 10 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:03.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:03 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:04.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:04 smithi156 ceph-mon[65097]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-04-24T16:51:04.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:04 smithi070 ceph-mon[81602]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-04-24T16:51:04.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:04 smithi070 ceph-mon[86496]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-04-24T16:51:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:05 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:05 smithi156 ceph-mon[65097]: pgmap v221: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 72/627 objects degraded (11.483%) 2024-04-24T16:51:05.411 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:51:05.226+0000 7f1c2706c7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-24T16:51:05.411 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:51:05.250+0000 7f1c2706c7c0 -1 osd.5 137 log_to_monitors true 2024-04-24T16:51:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:05 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:05 smithi070 ceph-mon[81602]: pgmap v221: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 72/627 objects degraded (11.483%) 2024-04-24T16:51:05.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:05 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:05.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:05 smithi070 ceph-mon[86496]: pgmap v221: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 72/627 objects degraded (11.483%) 2024-04-24T16:51:06.410 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:51:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:51:06.137+0000 7f1c184c4700 -1 osd.5 137 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:51:06.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:06 smithi156 ceph-mon[65097]: from='osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:51:06.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:06 smithi156 ceph-mon[65097]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:51:06.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:06 smithi070 ceph-mon[86496]: from='osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:51:06.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:06 smithi070 ceph-mon[86496]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:51:06.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:06 smithi070 ceph-mon[81602]: from='osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:51:06.520 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:06 smithi070 ceph-mon[81602]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T16:51:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:51:06] "GET /metrics HTTP/1.1" 200 38271 "" "Prometheus/2.43.0" 2024-04-24T16:51:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:07 smithi156 ceph-mon[65097]: pgmap v222: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 72/627 objects degraded (11.483%) 2024-04-24T16:51:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:07 smithi156 ceph-mon[65097]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T16:51:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:07 smithi156 ceph-mon[65097]: from='osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:07 smithi156 ceph-mon[65097]: osdmap e140: 8 total, 7 up, 8 in 2024-04-24T16:51:07.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:07 smithi156 ceph-mon[65097]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:07.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:07 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:07.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:07 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:07.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[81602]: pgmap v222: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 72/627 objects degraded (11.483%) 2024-04-24T16:51:07.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[81602]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T16:51:07.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[81602]: from='osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[81602]: osdmap e140: 8 total, 7 up, 8 in 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[81602]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[86496]: pgmap v222: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 72/627 objects degraded (11.483%) 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[86496]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[86496]: from='osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[86496]: osdmap e140: 8 total, 7 up, 8 in 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[86496]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:07.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:07 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:08.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:08 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:51:08.346 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:08 smithi156 ceph-mon[65097]: osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761] boot 2024-04-24T16:51:08.346 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:08 smithi156 ceph-mon[65097]: osdmap e141: 8 total, 8 up, 8 in 2024-04-24T16:51:08.346 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:08 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:51:08.346 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:08 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:08.346 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:08 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:08.346 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:08 smithi156 ceph-mon[65097]: osdmap e142: 8 total, 8 up, 8 in 2024-04-24T16:51:08.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:51:08.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[81602]: osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761] boot 2024-04-24T16:51:08.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[81602]: osdmap e141: 8 total, 8 up, 8 in 2024-04-24T16:51:08.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:51:08.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[81602]: osdmap e142: 8 total, 8 up, 8 in 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[86496]: osd.5 [v2:172.21.15.156:6808/2925761,v1:172.21.15.156:6809/2925761] boot 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[86496]: osdmap e141: 8 total, 8 up, 8 in 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:08.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:08 smithi070 ceph-mon[86496]: osdmap e142: 8 total, 8 up, 8 in 2024-04-24T16:51:09.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:09 smithi156 ceph-mon[65097]: pgmap v225: 161 pgs: 18 peering, 16 active+undersized+degraded, 25 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 51/627 objects degraded (8.134%) 2024-04-24T16:51:09.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:09 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 51/627 objects degraded (8.134%), 16 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:09.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:09 smithi070 ceph-mon[86496]: pgmap v225: 161 pgs: 18 peering, 16 active+undersized+degraded, 25 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 51/627 objects degraded (8.134%) 2024-04-24T16:51:09.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:09 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 51/627 objects degraded (8.134%), 16 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:09.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:09 smithi070 ceph-mon[81602]: pgmap v225: 161 pgs: 18 peering, 16 active+undersized+degraded, 25 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 51/627 objects degraded (8.134%) 2024-04-24T16:51:09.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:09 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 51/627 objects degraded (8.134%), 16 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:11.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: pgmap v227: 161 pgs: 5 activating, 18 peering, 13 active+undersized+degraded, 23 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-24T16:51:11.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:11.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:51:11.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:11.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:11 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:11.543 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:51:11.100+0000 7f22938a6700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: pgmap v227: 161 pgs: 5 activating, 18 peering, 13 active+undersized+degraded, 23 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:11.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: pgmap v227: 161 pgs: 5 activating, 18 peering, 13 active+undersized+degraded, 23 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:11.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:11 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:12 smithi156 ceph-mon[65097]: pgmap v228: 161 pgs: 5 activating, 18 peering, 10 active+undersized+degraded, 11 active+undersized, 117 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 35/627 objects degraded (5.582%) 2024-04-24T16:51:12.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:12 smithi156 ceph-mon[65097]: pgmap v229: 161 pgs: 5 activating, 18 peering, 10 active+undersized+degraded, 11 active+undersized, 117 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 35/627 objects degraded (5.582%) 2024-04-24T16:51:12.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:12 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:12.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:12 smithi156 ceph-mon[65097]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-24T16:51:12.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[81602]: pgmap v228: 161 pgs: 5 activating, 18 peering, 10 active+undersized+degraded, 11 active+undersized, 117 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 35/627 objects degraded (5.582%) 2024-04-24T16:51:12.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[81602]: pgmap v229: 161 pgs: 5 activating, 18 peering, 10 active+undersized+degraded, 11 active+undersized, 117 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 35/627 objects degraded (5.582%) 2024-04-24T16:51:12.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:12.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[81602]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-24T16:51:12.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[86496]: pgmap v228: 161 pgs: 5 activating, 18 peering, 10 active+undersized+degraded, 11 active+undersized, 117 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 35/627 objects degraded (5.582%) 2024-04-24T16:51:12.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[86496]: pgmap v229: 161 pgs: 5 activating, 18 peering, 10 active+undersized+degraded, 11 active+undersized, 117 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 35/627 objects degraded (5.582%) 2024-04-24T16:51:12.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:12.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:12 smithi070 ceph-mon[86496]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-24T16:51:13.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:13 smithi070 ceph-mon[81602]: pgmap v230: 161 pgs: 5 activating, 11 peering, 145 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-04-24T16:51:13.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:13 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/627 objects degraded (5.582%), 10 pgs degraded) 2024-04-24T16:51:13.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:13 smithi070 ceph-mon[86496]: pgmap v230: 161 pgs: 5 activating, 11 peering, 145 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-04-24T16:51:13.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:13 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/627 objects degraded (5.582%), 10 pgs degraded) 2024-04-24T16:51:13.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:13 smithi156 ceph-mon[65097]: pgmap v230: 161 pgs: 5 activating, 11 peering, 145 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-04-24T16:51:13.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:13 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/627 objects degraded (5.582%), 10 pgs degraded) 2024-04-24T16:51:14.444 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:14 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:14.445 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:14 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:14.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:14 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:15.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:15 smithi070 ceph-mon[86496]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-04-24T16:51:15.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:15 smithi070 ceph-mon[81602]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-04-24T16:51:15.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:15 smithi156 ceph-mon[65097]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-04-24T16:51:15.856 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:51:16.514 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:16 smithi070 ceph-mon[86496]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:16.514 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:51:16.514 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (7m) 80s ago 29m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:51:16.514 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (7m) 10s ago 29m 77.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:51:16.514 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 80s ago 29m 64.0M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:51:16.514 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (9m) 10s ago 34m 454M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (19m) 80s ago 35m 509M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (5m) 80s ago 35m 51.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (6m) 10s ago 34m 48.0M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (5m) 80s ago 34m 44.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (8m) 80s ago 30m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (8m) 10s ago 30m 16.5M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (3m) 80s ago 33m 52.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (3m) 80s ago 33m 54.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (2m) 80s ago 33m 44.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (85s) 80s ago 32m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (49s) 10s ago 32m 51.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (14s) 10s ago 32m 15.1M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (31m) 10s ago 31m 55.7M 3248M 17.2.0 e1d6a67b021e 5d22c33c1226 2024-04-24T16:51:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (31m) 10s ago 31m 60.6M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:51:16.516 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (8m) 10s ago 30m 69.3M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:51:16.516 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (29m) 80s ago 29m 78.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:51:16.516 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (29m) 10s ago 29m 78.1M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:51:16.516 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:16 smithi070 ceph-mon[81602]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:16.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:16 smithi156 ceph-mon[65097]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:51:16] "GET /metrics HTTP/1.1" 200 38271 "" "Prometheus/2.43.0" 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:16.911 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 6 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 11 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:51:16.912 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:51:17.251 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:51:17.251 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:51:17.251 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:51:17.251 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-24T16:51:17.251 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:51:17.251 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "6/8 daemons upgraded", 2024-04-24T16:51:17.251 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-04-24T16:51:17.252 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:51:17.252 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:51:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[81602]: from='client.54540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[81602]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2409717613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[81602]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:51:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:17.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:17.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[86496]: from='client.54540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:17.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[86496]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:17.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2409717613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:17.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[86496]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:51:17.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:17.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:17 smithi156 ceph-mon[65097]: from='client.54540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:17 smithi156 ceph-mon[65097]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:17 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2409717613' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:17 smithi156 ceph-mon[65097]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:51:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:17.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:18 smithi070 ceph-mon[81602]: from='client.64495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:18 smithi070 ceph-mon[86496]: from='client.64495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:18.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:18 smithi156 ceph-mon[65097]: from='client.64495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:19.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:19 smithi070 ceph-mon[81602]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:51:19.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:19 smithi070 ceph-mon[86496]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:51:19.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:19 smithi156 ceph-mon[65097]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:51:22.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:22 smithi070 ceph-mon[81602]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-24T16:51:22.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:22 smithi070 ceph-mon[86496]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-24T16:51:22.364 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:22 smithi156 ceph-mon[65097]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-24T16:51:23.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:23 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:23.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:23 smithi070 ceph-mon[86496]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:23.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:23 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:23.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:23 smithi070 ceph-mon[81602]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:23 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:23.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:23 smithi156 ceph-mon[65097]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:24.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:24 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:24.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:24 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:24.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:24 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:25.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:25 smithi070 ceph-mon[81602]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:25.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:25 smithi070 ceph-mon[86496]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:25.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:25 smithi156 ceph-mon[65097]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:26.693 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:26 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:26.693 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:26 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:26.694 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:26 smithi156 ceph-mon[65097]: Upgrade: osd.6 is safe to restart 2024-04-24T16:51:26.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:51:26] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.43.0" 2024-04-24T16:51:26.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:26 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:26.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:26 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:26.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:26 smithi070 ceph-mon[81602]: Upgrade: osd.6 is safe to restart 2024-04-24T16:51:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:26 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:26 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-24T16:51:26.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:26 smithi070 ceph-mon[86496]: Upgrade: osd.6 is safe to restart 2024-04-24T16:51:27.697 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:27 smithi156 ceph-mon[65097]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:51:27.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:27 smithi070 ceph-mon[86496]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:51:27.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:27 smithi070 ceph-mon[81602]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:51:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:28 smithi156 ceph-mon[65097]: Upgrade: Updating osd.6 2024-04-24T16:51:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:51:28.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:28.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:28 smithi156 ceph-mon[65097]: Deploying daemon osd.6 on smithi156 2024-04-24T16:51:29.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[81602]: Upgrade: Updating osd.6 2024-04-24T16:51:29.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[81602]: Deploying daemon osd.6 on smithi156 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[86496]: Upgrade: Updating osd.6 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:29.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:28 smithi070 ceph-mon[86496]: Deploying daemon osd.6 on smithi156 2024-04-24T16:51:29.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:29 smithi156 ceph-mon[65097]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:30.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:29 smithi070 ceph-mon[86496]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:29 smithi070 ceph-mon[81602]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:51:31.137 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:30 smithi156 systemd[1]: Stopping Ceph osd.6 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:51:31.410 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[31875]: 2024-04-24T16:51:31.136+0000 7f91d953a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:51:31.410 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[31875]: 2024-04-24T16:51:31.136+0000 7f91d953a700 -1 osd.6 142 *** Got signal Terminated *** 2024-04-24T16:51:31.410 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:31 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[31875]: 2024-04-24T16:51:31.136+0000 7f91d953a700 -1 osd.6 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:51:32.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:32 smithi070 ceph-mon[81602]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:51:32.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:32 smithi070 ceph-mon[81602]: osd.6 marked itself down and dead 2024-04-24T16:51:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:32 smithi070 ceph-mon[86496]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:51:32.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:32 smithi070 ceph-mon[86496]: osd.6 marked itself down and dead 2024-04-24T16:51:32.382 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:32 smithi156 ceph-mon[65097]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:51:32.382 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:32 smithi156 ceph-mon[65097]: osd.6 marked itself down and dead 2024-04-24T16:51:32.660 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:32 smithi156 podman[85252]: 2024-04-24 16:51:32.382522362 +0000 UTC m=+1.334002678 container died 5d22c33c12262f18d68447ed1cc5f560f3097b1c0fca79a995d0528d0b73c913 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.openshift.expose-services=, version=8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-04-24T16:51:32.987 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:32 smithi156 podman[85252]: 2024-04-24 16:51:32.753333236 +0000 UTC m=+1.704813561 container cleanup 5d22c33c12262f18d68447ed1cc5f560f3097b1c0fca79a995d0528d0b73c913 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, version=8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_CLEAN=True) 2024-04-24T16:51:32.987 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:32 smithi156 bash[85252]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6 2024-04-24T16:51:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:51:33.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: osdmap e143: 8 total, 7 up, 8 in 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [4, 2]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.5", "id": [3, 1]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 6]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [5, 6]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 6]}]: dispatch 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: osdmap e143: 8 total, 7 up, 8 in 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [4, 2]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.5", "id": [3, 1]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 6]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-04-24T16:51:33.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [5, 6]}]: dispatch 2024-04-24T16:51:33.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 6]}]: dispatch 2024-04-24T16:51:33.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:33.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:33.344 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:32 smithi156 podman[85270]: 2024-04-24 16:51:32.987931365 +0000 UTC m=+0.607967939 container remove 5d22c33c12262f18d68447ed1cc5f560f3097b1c0fca79a995d0528d0b73c913 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.component=centos-stream-container, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, version=8, vcs-type=git, RELEASE=HEAD, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-24T16:51:33.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:51:33.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: osdmap e143: 8 total, 7 up, 8 in 2024-04-24T16:51:33.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [4, 2]}]: dispatch 2024-04-24T16:51:33.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]: dispatch 2024-04-24T16:51:33.344 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.5", "id": [3, 1]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 6]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [5, 6]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 6]}]: dispatch 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:33.345 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:33.595 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:33 smithi156 podman[85363]: 2024-04-24 16:51:33.247488348 +0000 UTC m=+0.018798994 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:51:33.595 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:33 smithi156 podman[85363]: 2024-04-24 16:51:33.372159063 +0000 UTC m=+0.143469751 container create 699460957e73e3d4d2fd27fea15230d041ecd953ad740acf06965730295fc7fc (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-24T16:51:33.595 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:33 smithi156 podman[85363]: 2024-04-24 16:51:33.592090945 +0000 UTC m=+0.363401601 container init 699460957e73e3d4d2fd27fea15230d041ecd953ad740acf06965730295fc7fc (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-24T16:51:33.595 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:33 smithi156 podman[85363]: 2024-04-24 16:51:33.595610078 +0000 UTC m=+0.366920730 container start 699460957e73e3d4d2fd27fea15230d041ecd953ad740acf06965730295fc7fc (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:51:33.886 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:33 smithi156 podman[85363]: 2024-04-24 16:51:33.683650966 +0000 UTC m=+0.454961631 container attach 699460957e73e3d4d2fd27fea15230d041ecd953ad740acf06965730295fc7fc (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-24T16:51:34.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:33 smithi156 podman[85363]: 2024-04-24 16:51:33.887560661 +0000 UTC m=+0.658871308 container died 699460957e73e3d4d2fd27fea15230d041ecd953ad740acf06965730295fc7fc (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: pgmap v241: 161 pgs: 4 peering, 13 stale+active+clean, 144 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [4, 2]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.5", "id": [3, 1]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 6]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [5, 6]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 6]}]': finished 2024-04-24T16:51:34.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:34 smithi156 ceph-mon[65097]: osdmap e144: 8 total, 7 up, 8 in 2024-04-24T16:51:34.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: pgmap v241: 161 pgs: 4 peering, 13 stale+active+clean, 144 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [4, 2]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.5", "id": [3, 1]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 6]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [5, 6]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 6]}]': finished 2024-04-24T16:51:34.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[81602]: osdmap e144: 8 total, 7 up, 8 in 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: pgmap v241: 161 pgs: 4 peering, 13 stale+active+clean, 144 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [4, 2]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.12", "id": [1, 6]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [7, 6]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.5", "id": [3, 1]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 6]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 0]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [5, 6]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1a", "id": [5, 6]}]': finished 2024-04-24T16:51:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:34 smithi070 ceph-mon[86496]: osdmap e144: 8 total, 7 up, 8 in 2024-04-24T16:51:34.905 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:34 smithi156 podman[85389]: 2024-04-24 16:51:34.470785399 +0000 UTC m=+0.574390830 container remove 699460957e73e3d4d2fd27fea15230d041ecd953ad740acf06965730295fc7fc (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-24T16:51:34.905 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:34 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-24T16:51:34.905 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:34 smithi156 systemd[1]: Stopped Ceph osd.6 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:51:34.905 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:34 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.6.service: Consumed 15.023s CPU time. 2024-04-24T16:51:34.905 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:34 smithi156 systemd[1]: Starting Ceph osd.6 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:51:35.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:35.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-mon[65097]: osdmap e145: 8 total, 7 up, 8 in 2024-04-24T16:51:35.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-mon[65097]: osdmap e146: 8 total, 7 up, 8 in 2024-04-24T16:51:35.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:34 smithi156 podman[85490]: 2024-04-24 16:51:34.808577991 +0000 UTC m=+0.019005143 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:51:35.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:34 smithi156 podman[85490]: 2024-04-24 16:51:34.921693139 +0000 UTC m=+0.132120280 container create 607f6f770ac032c345ede4d08abbbea05dfd23c921b073df384bcb2075c170cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-24T16:51:35.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 podman[85490]: 2024-04-24 16:51:35.132985174 +0000 UTC m=+0.343412313 container init 607f6f770ac032c345ede4d08abbbea05dfd23c921b073df384bcb2075c170cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-24T16:51:35.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 podman[85490]: 2024-04-24 16:51:35.136438248 +0000 UTC m=+0.346865387 container start 607f6f770ac032c345ede4d08abbbea05dfd23c921b073df384bcb2075c170cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:51:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:35 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:35 smithi070 ceph-mon[81602]: osdmap e145: 8 total, 7 up, 8 in 2024-04-24T16:51:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:35 smithi070 ceph-mon[81602]: osdmap e146: 8 total, 7 up, 8 in 2024-04-24T16:51:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:35 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:35.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:35 smithi070 ceph-mon[86496]: osdmap e145: 8 total, 7 up, 8 in 2024-04-24T16:51:35.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:35 smithi070 ceph-mon[86496]: osdmap e146: 8 total, 7 up, 8 in 2024-04-24T16:51:35.621 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 podman[85490]: 2024-04-24 16:51:35.241598495 +0000 UTC m=+0.452025645 container attach 607f6f770ac032c345ede4d08abbbea05dfd23c921b073df384bcb2075c170cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-24T16:51:35.910 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 bash[85490]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-24T16:51:35.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-24T16:51:35.912 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T16:51:35.912 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate[85505]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-24T16:51:35.912 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:35 smithi156 podman[85490]: 2024-04-24 16:51:35.647202255 +0000 UTC m=+0.857629386 container died 607f6f770ac032c345ede4d08abbbea05dfd23c921b073df384bcb2075c170cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-24T16:51:36.228 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:36 smithi156 ceph-mon[65097]: pgmap v244: 161 pgs: 2 unknown, 1 active+undersized+degraded, 13 peering, 10 stale+active+clean, 2 active+undersized, 133 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-04-24T16:51:36.228 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:36 smithi156 ceph-mon[65097]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:51:36.228 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:36 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 2/627 objects degraded (0.319%), 1 pg degraded (PG_DEGRADED) 2024-04-24T16:51:36.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:36 smithi070 ceph-mon[86496]: pgmap v244: 161 pgs: 2 unknown, 1 active+undersized+degraded, 13 peering, 10 stale+active+clean, 2 active+undersized, 133 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-04-24T16:51:36.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:36 smithi070 ceph-mon[86496]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:51:36.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:36 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 2/627 objects degraded (0.319%), 1 pg degraded (PG_DEGRADED) 2024-04-24T16:51:36.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:36 smithi070 ceph-mon[81602]: pgmap v244: 161 pgs: 2 unknown, 1 active+undersized+degraded, 13 peering, 10 stale+active+clean, 2 active+undersized, 133 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-04-24T16:51:36.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:36 smithi070 ceph-mon[81602]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:51:36.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:36 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 2/627 objects degraded (0.319%), 1 pg degraded (PG_DEGRADED) 2024-04-24T16:51:36.563 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 podman[85632]: 2024-04-24 16:51:36.228884178 +0000 UTC m=+0.576325326 container remove 607f6f770ac032c345ede4d08abbbea05dfd23c921b073df384bcb2075c170cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:51:36.563 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 podman[85699]: 2024-04-24 16:51:36.34600275 +0000 UTC m=+0.018383222 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:51:36.563 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 podman[85699]: 2024-04-24 16:51:36.463070314 +0000 UTC m=+0.135450757 container create 2972333b52bf6b9962c3522b473fb6b20b7427f34235838f3db6b74cab712ffe (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=) 2024-04-24T16:51:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:51:36] "GET /metrics HTTP/1.1" 200 38278 "" "Prometheus/2.43.0" 2024-04-24T16:51:36.910 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 podman[85699]: 2024-04-24 16:51:36.648556358 +0000 UTC m=+0.320936802 container init 2972333b52bf6b9962c3522b473fb6b20b7427f34235838f3db6b74cab712ffe (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T16:51:36.910 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 podman[85699]: 2024-04-24 16:51:36.651777684 +0000 UTC m=+0.324158128 container start 2972333b52bf6b9962c3522b473fb6b20b7427f34235838f3db6b74cab712ffe (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T16:51:36.910 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 ceph-osd[85717]: -- 172.21.15.156:0/1134001807 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56014ce61680 con 0x56014ce2ac00 2024-04-24T16:51:36.910 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 bash[85699]: 2972333b52bf6b9962c3522b473fb6b20b7427f34235838f3db6b74cab712ffe 2024-04-24T16:51:36.910 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:36 smithi156 systemd[1]: Started Ceph osd.6 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:51:37.631 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:37 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:51:37.195+0000 7f57083747c0 -1 Falling back to public interface 2024-04-24T16:51:37.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:37.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:37.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:37 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:37.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:37 smithi156 ceph-mon[65097]: pgmap v246: 161 pgs: 4 active+clean+remapped, 103 active+clean, 25 active+undersized, 13 peering, 14 active+undersized+degraded, 2 unknown; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:38.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:38.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:38.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:38.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[81602]: pgmap v246: 161 pgs: 4 active+clean+remapped, 103 active+clean, 25 active+undersized, 13 peering, 14 active+undersized+degraded, 2 unknown; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:38.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:38.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:38.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:38.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:37 smithi070 ceph-mon[86496]: pgmap v246: 161 pgs: 4 active+clean+remapped, 103 active+clean, 25 active+undersized, 13 peering, 14 active+undersized+degraded, 2 unknown; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 50/627 objects degraded (7.974%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:39.352 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:39 smithi156 ceph-mon[65097]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-24T16:51:39.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:39 smithi070 ceph-mon[86496]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-24T16:51:39.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:39 smithi070 ceph-mon[81602]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-24T16:51:40.172 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:39 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:51:39.916+0000 7f57083747c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-24T16:51:40.172 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:39 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:51:39.947+0000 7f57083747c0 -1 osd.6 142 log_to_monitors true 2024-04-24T16:51:40.172 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:51:40 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:51:40.089+0000 7f56f97cc700 -1 osd.6 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:51:40.172 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:40 smithi156 ceph-mon[65097]: pgmap v247: 161 pgs: 8 active+clean+remapped, 102 active+clean, 31 active+undersized, 20 active+undersized+degraded; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:40.172 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:40 smithi156 ceph-mon[65097]: from='osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:51:40.172 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:40 smithi156 ceph-mon[65097]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:51:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:40 smithi070 ceph-mon[86496]: pgmap v247: 161 pgs: 8 active+clean+remapped, 102 active+clean, 31 active+undersized, 20 active+undersized+degraded; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:40 smithi070 ceph-mon[86496]: from='osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:51:40.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:40 smithi070 ceph-mon[86496]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:51:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:40 smithi070 ceph-mon[81602]: pgmap v247: 161 pgs: 8 active+clean+remapped, 102 active+clean, 31 active+undersized, 20 active+undersized+degraded; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:40 smithi070 ceph-mon[81602]: from='osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:51:40.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:40 smithi070 ceph-mon[81602]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T16:51:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:41 smithi156 ceph-mon[65097]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T16:51:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:41 smithi156 ceph-mon[65097]: from='osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:41 smithi156 ceph-mon[65097]: osdmap e147: 8 total, 7 up, 8 in 2024-04-24T16:51:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:41 smithi156 ceph-mon[65097]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:41.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[86496]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T16:51:41.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[86496]: from='osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:41.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[86496]: osdmap e147: 8 total, 7 up, 8 in 2024-04-24T16:51:41.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[86496]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:41.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[81602]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T16:51:41.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[81602]: from='osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:41.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[81602]: osdmap e147: 8 total, 7 up, 8 in 2024-04-24T16:51:41.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:41 smithi070 ceph-mon[81602]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:51:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: pgmap v249: 161 pgs: 8 active+clean+remapped, 102 active+clean, 31 active+undersized, 20 active+undersized+degraded; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:51:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996] boot 2024-04-24T16:51:42.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: osdmap e148: 8 total, 8 up, 8 in 2024-04-24T16:51:42.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:51:42.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:42.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:42.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:42 smithi156 ceph-mon[65097]: osdmap e149: 8 total, 8 up, 8 in 2024-04-24T16:51:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: pgmap v249: 161 pgs: 8 active+clean+remapped, 102 active+clean, 31 active+undersized, 20 active+undersized+degraded; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:51:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996] boot 2024-04-24T16:51:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: osdmap e148: 8 total, 8 up, 8 in 2024-04-24T16:51:42.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[81602]: osdmap e149: 8 total, 8 up, 8 in 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: pgmap v249: 161 pgs: 8 active+clean+remapped, 102 active+clean, 31 active+undersized, 20 active+undersized+degraded; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: osd.6 [v2:172.21.15.156:6816/3036337996,v1:172.21.15.156:6817/3036337996] boot 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: osdmap e148: 8 total, 8 up, 8 in 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:42.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:42 smithi070 ceph-mon[86496]: osdmap e149: 8 total, 8 up, 8 in 2024-04-24T16:51:43.138 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:43 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 20 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:43.138 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:43.138 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:43 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:43.138 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:43 smithi156 ceph-mon[65097]: osdmap e150: 8 total, 8 up, 8 in 2024-04-24T16:51:43.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 20 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:43.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:43.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:43.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[81602]: osdmap e150: 8 total, 8 up, 8 in 2024-04-24T16:51:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 76/627 objects degraded (12.121%), 20 pgs degraded (PG_DEGRADED) 2024-04-24T16:51:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:43.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:43 smithi070 ceph-mon[86496]: osdmap e150: 8 total, 8 up, 8 in 2024-04-24T16:51:44.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:44 smithi156 ceph-mon[65097]: pgmap v252: 161 pgs: 1 remapped+peering, 7 peering, 7 active+clean+remapped, 106 active+clean, 24 active+undersized, 16 active+undersized+degraded; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 56/627 objects degraded (8.931%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:44.444 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:44 smithi070 ceph-mon[86496]: pgmap v252: 161 pgs: 1 remapped+peering, 7 peering, 7 active+clean+remapped, 106 active+clean, 24 active+undersized, 16 active+undersized+degraded; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 56/627 objects degraded (8.931%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:44.445 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:44 smithi070 ceph-mon[81602]: pgmap v252: 161 pgs: 1 remapped+peering, 7 peering, 7 active+clean+remapped, 106 active+clean, 24 active+undersized, 16 active+undersized+degraded; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 56/627 objects degraded (8.931%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:45 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:45.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:45 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:45.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:45 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:46.293 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:45 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:51:45.961+0000 7f22938a6700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-24T16:51:46.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: pgmap v254: 161 pgs: 1 activating, 1 remapped+peering, 8 peering, 4 active+clean+remapped, 115 active+clean, 20 active+undersized, 12 active+undersized+degraded; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/627 objects degraded (6.380%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:51:46.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: pgmap v254: 161 pgs: 1 activating, 1 remapped+peering, 8 peering, 4 active+clean+remapped, 115 active+clean, 20 active+undersized, 12 active+undersized+degraded; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/627 objects degraded (6.380%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:51:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: pgmap v254: 161 pgs: 1 activating, 1 remapped+peering, 8 peering, 4 active+clean+remapped, 115 active+clean, 20 active+undersized, 12 active+undersized+degraded; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/627 objects degraded (6.380%); 1/627 objects misplaced (0.159%) 2024-04-24T16:51:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:51:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:51:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:46.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:51:46.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:46.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:46 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:51:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:51:46] "GET /metrics HTTP/1.1" 200 38278 "" "Prometheus/2.43.0" 2024-04-24T16:51:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:47 smithi156 ceph-mon[65097]: pgmap v255: 161 pgs: 1 activating, 8 peering, 152 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-04-24T16:51:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:47 smithi156 ceph-mon[65097]: pgmap v256: 161 pgs: 1 activating, 8 peering, 152 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 4 B/s, 0 objects/s recovering 2024-04-24T16:51:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:47 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:51:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:47 smithi156 ceph-mon[65097]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-24T16:51:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:47 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 12 pgs degraded) 2024-04-24T16:51:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:47.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[81602]: pgmap v255: 161 pgs: 1 activating, 8 peering, 152 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-04-24T16:51:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[81602]: pgmap v256: 161 pgs: 1 activating, 8 peering, 152 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 4 B/s, 0 objects/s recovering 2024-04-24T16:51:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:51:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[81602]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-24T16:51:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 12 pgs degraded) 2024-04-24T16:51:47.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[86496]: pgmap v255: 161 pgs: 1 activating, 8 peering, 152 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[86496]: pgmap v256: 161 pgs: 1 activating, 8 peering, 152 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s; 4 B/s, 0 objects/s recovering 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[86496]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 12 pgs degraded) 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:47.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:51:47.612 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:51:48.267 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:51:48.267 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (8m) 112s ago 30m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:51:48.267 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (8m) 7s ago 30m 79.8M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:51:48.267 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 112s ago 29m 64.0M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (9m) 7s ago 34m 454M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (19m) 112s ago 36m 509M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (6m) 112s ago 36m 51.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (7m) 7s ago 35m 49.6M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (6m) 112s ago 35m 44.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (9m) 112s ago 31m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (9m) 7s ago 31m 15.7M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (4m) 112s ago 34m 52.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (3m) 112s ago 34m 54.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (2m) 112s ago 33m 44.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (117s) 112s ago 33m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (81s) 7s ago 33m 52.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (46s) 7s ago 32m 46.5M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:51:48.268 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (11s) 7s ago 32m 14.5M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:51:48.269 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (31m) 7s ago 31m 62.2M 3248M 17.2.0 e1d6a67b021e 8dda71f02797 2024-04-24T16:51:48.269 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (8m) 7s ago 31m 69.3M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:51:48.269 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (29m) 112s ago 29m 78.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:51:48.269 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (29m) 7s ago 29m 78.3M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:51:48.663 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:51:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:51:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:51:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:51:48.663 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 7 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 12 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:51:48.664 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "7/8 daemons upgraded", 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:51:48.996 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:51:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[86496]: from='client.64504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[86496]: pgmap v257: 161 pgs: 1 activating, 1 peering, 159 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-04-24T16:51:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[86496]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:49.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/649491249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:49.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[81602]: from='client.64504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:49.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[81602]: pgmap v257: 161 pgs: 1 activating, 1 peering, 159 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-04-24T16:51:49.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[81602]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:49.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:49 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/649491249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:49 smithi156 ceph-mon[65097]: from='client.64504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:49 smithi156 ceph-mon[65097]: pgmap v257: 161 pgs: 1 activating, 1 peering, 159 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-04-24T16:51:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:49 smithi156 ceph-mon[65097]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:49.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:49 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/649491249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:51:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:50 smithi156 ceph-mon[65097]: from='client.54579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:50 smithi156 ceph-mon[65097]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:50.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:50 smithi070 ceph-mon[86496]: from='client.54579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:50.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:50 smithi070 ceph-mon[86496]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:50.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:50 smithi070 ceph-mon[81602]: from='client.54579 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:50.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:50 smithi070 ceph-mon[81602]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:51:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:51 smithi156 ceph-mon[65097]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:51.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:51 smithi070 ceph-mon[81602]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:51.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:51 smithi070 ceph-mon[86496]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:53 smithi156 ceph-mon[65097]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:53 smithi070 ceph-mon[86496]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:53.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:53 smithi070 ceph-mon[81602]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:55 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:55 smithi156 ceph-mon[65097]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:55.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:55 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:55.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:55 smithi070 ceph-mon[86496]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:55 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:51:55.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:55 smithi070 ceph-mon[81602]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-24T16:51:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:51:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:51:56] "GET /metrics HTTP/1.1" 200 38291 "" "Prometheus/2.43.0" 2024-04-24T16:51:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:57 smithi156 ceph-mon[65097]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:51:57.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:57 smithi070 ceph-mon[86496]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:51:57.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:57 smithi070 ceph-mon[81602]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:51:58.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:58 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:58.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:51:58 smithi070 ceph-mon[86496]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:51:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:58 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:58.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:51:58 smithi070 ceph-mon[81602]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:51:58.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:58 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:51:58.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:51:58 smithi156 ceph-mon[65097]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:01.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:00 smithi156 ceph-mon[65097]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:01.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:00 smithi070 ceph-mon[86496]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:01.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:00 smithi070 ceph-mon[81602]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:52:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:01 smithi156 ceph-mon[65097]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:52:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:01 smithi156 ceph-mon[65097]: Upgrade: osd.7 is safe to restart 2024-04-24T16:52:02.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:52:02.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:01 smithi070 ceph-mon[86496]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:52:02.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:01 smithi070 ceph-mon[86496]: Upgrade: osd.7 is safe to restart 2024-04-24T16:52:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:52:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:01 smithi070 ceph-mon[81602]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-24T16:52:02.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:01 smithi070 ceph-mon[81602]: Upgrade: osd.7 is safe to restart 2024-04-24T16:52:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:02 smithi156 ceph-mon[65097]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:52:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:03.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[86496]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:03.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:03.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:03.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:52:03.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:03.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[81602]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:03.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:03.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:03.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T16:52:03.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:04.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:03 smithi156 ceph-mon[65097]: Upgrade: Updating osd.7 2024-04-24T16:52:04.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:03 smithi156 ceph-mon[65097]: Deploying daemon osd.7 on smithi156 2024-04-24T16:52:04.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:03 smithi070 ceph-mon[86496]: Upgrade: Updating osd.7 2024-04-24T16:52:04.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:03 smithi070 ceph-mon[86496]: Deploying daemon osd.7 on smithi156 2024-04-24T16:52:04.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:03 smithi070 ceph-mon[81602]: Upgrade: Updating osd.7 2024-04-24T16:52:04.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:03 smithi070 ceph-mon[81602]: Deploying daemon osd.7 on smithi156 2024-04-24T16:52:05.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:04 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:05.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:04 smithi156 ceph-mon[65097]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-04-24T16:52:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:04 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:05.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:04 smithi070 ceph-mon[86496]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-04-24T16:52:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:04 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:05.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:04 smithi070 ceph-mon[81602]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 22 op/s 2024-04-24T16:52:06.151 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:05 smithi156 systemd[1]: Stopping Ceph osd.7 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:52:06.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[34873]: 2024-04-24T16:52:06.191+0000 7fe1ec329700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:52:06.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[34873]: 2024-04-24T16:52:06.191+0000 7fe1ec329700 -1 osd.7 150 *** Got signal Terminated *** 2024-04-24T16:52:06.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[34873]: 2024-04-24T16:52:06.191+0000 7fe1ec329700 -1 osd.7 150 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:52:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:52:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:52:06] "GET /metrics HTTP/1.1" 200 38288 "" "Prometheus/2.43.0" 2024-04-24T16:52:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:06 smithi156 ceph-mon[65097]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2024-04-24T16:52:07.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:06 smithi156 ceph-mon[65097]: osd.7 marked itself down and dead 2024-04-24T16:52:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:06 smithi070 ceph-mon[86496]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2024-04-24T16:52:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:06 smithi070 ceph-mon[86496]: osd.7 marked itself down and dead 2024-04-24T16:52:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:06 smithi070 ceph-mon[81602]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 46 op/s 2024-04-24T16:52:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:06 smithi070 ceph-mon[81602]: osd.7 marked itself down and dead 2024-04-24T16:52:07.576 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:07 smithi156 podman[90207]: 2024-04-24 16:52:07.301603239 +0000 UTC m=+1.233889761 container died 8dda71f02797ac2046fd3dbe5b4ddcb4fb60d0972f3b1f6de427c54daeb4f2b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, distribution-scope=public, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.expose-services=, ceph=True, RELEASE=HEAD) 2024-04-24T16:52:07.888 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:07 smithi156 podman[90207]: 2024-04-24 16:52:07.675085054 +0000 UTC m=+1.607371584 container cleanup 8dda71f02797ac2046fd3dbe5b4ddcb4fb60d0972f3b1f6de427c54daeb4f2b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, name=centos-stream, io.buildah.version=1.19.8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, distribution-scope=public, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-24T16:52:07.888 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:07 smithi156 bash[90207]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7 2024-04-24T16:52:08.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:07 smithi156 ceph-mon[65097]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:52:08.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:07 smithi156 ceph-mon[65097]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-24T16:52:08.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:07 smithi156 ceph-mon[65097]: osdmap e151: 8 total, 7 up, 8 in 2024-04-24T16:52:08.160 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:07 smithi156 podman[90236]: 2024-04-24 16:52:07.960065879 +0000 UTC m=+0.653815962 container remove 8dda71f02797ac2046fd3dbe5b4ddcb4fb60d0972f3b1f6de427c54daeb4f2b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-type=git, RELEASE=HEAD, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870) 2024-04-24T16:52:08.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:07 smithi070 ceph-mon[86496]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:52:08.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:07 smithi070 ceph-mon[86496]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-24T16:52:08.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:07 smithi070 ceph-mon[86496]: osdmap e151: 8 total, 7 up, 8 in 2024-04-24T16:52:08.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:07 smithi070 ceph-mon[81602]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-24T16:52:08.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:07 smithi070 ceph-mon[81602]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-24T16:52:08.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:07 smithi070 ceph-mon[81602]: osdmap e151: 8 total, 7 up, 8 in 2024-04-24T16:52:08.571 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:08 smithi156 podman[90329]: 2024-04-24 16:52:08.175612338 +0000 UTC m=+0.018301212 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:52:08.571 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:08 smithi156 podman[90329]: 2024-04-24 16:52:08.294409293 +0000 UTC m=+0.137098150 container create 1e0f0bf91619a717e709c6f55ac8ee41f32642f503e19f20c31257868ec51d28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-24T16:52:08.571 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:08 smithi156 podman[90329]: 2024-04-24 16:52:08.505375152 +0000 UTC m=+0.348064006 container init 1e0f0bf91619a717e709c6f55ac8ee41f32642f503e19f20c31257868ec51d28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:52:08.572 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:08 smithi156 podman[90329]: 2024-04-24 16:52:08.508628597 +0000 UTC m=+0.351317460 container start 1e0f0bf91619a717e709c6f55ac8ee41f32642f503e19f20c31257868ec51d28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:52:08.889 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:08 smithi156 podman[90329]: 2024-04-24 16:52:08.572015184 +0000 UTC m=+0.414704048 container attach 1e0f0bf91619a717e709c6f55ac8ee41f32642f503e19f20c31257868ec51d28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-24T16:52:08.890 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:08 smithi156 podman[90329]: 2024-04-24 16:52:08.809033456 +0000 UTC m=+0.651722309 container died 1e0f0bf91619a717e709c6f55ac8ee41f32642f503e19f20c31257868ec51d28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:52:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:08 smithi156 ceph-mon[65097]: pgmap v268: 161 pgs: 4 peering, 21 stale+active+clean, 136 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-04-24T16:52:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:08 smithi156 ceph-mon[65097]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:52:09.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:08 smithi156 ceph-mon[65097]: osdmap e152: 8 total, 7 up, 8 in 2024-04-24T16:52:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:08 smithi070 ceph-mon[86496]: pgmap v268: 161 pgs: 4 peering, 21 stale+active+clean, 136 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-04-24T16:52:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:08 smithi070 ceph-mon[86496]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:52:09.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:08 smithi070 ceph-mon[86496]: osdmap e152: 8 total, 7 up, 8 in 2024-04-24T16:52:09.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:08 smithi070 ceph-mon[81602]: pgmap v268: 161 pgs: 4 peering, 21 stale+active+clean, 136 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 95 op/s 2024-04-24T16:52:09.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:08 smithi070 ceph-mon[81602]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-24T16:52:09.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:08 smithi070 ceph-mon[81602]: osdmap e152: 8 total, 7 up, 8 in 2024-04-24T16:52:09.860 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:09 smithi156 podman[90355]: 2024-04-24 16:52:09.425820725 +0000 UTC m=+0.608780187 container remove 1e0f0bf91619a717e709c6f55ac8ee41f32642f503e19f20c31257868ec51d28 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:52:09.860 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:09 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-24T16:52:09.860 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:09 smithi156 systemd[1]: Stopped Ceph osd.7 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:52:09.860 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:09 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.7.service: Consumed 16.661s CPU time. 2024-04-24T16:52:09.860 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:09 smithi156 systemd[1]: Starting Ceph osd.7 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:52:10.120 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:09 smithi156 ceph-mon[65097]: Health check failed: Degraded data redundancy: 9/627 objects degraded (1.435%), 4 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:10.120 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:09 smithi156 podman[90459]: 2024-04-24 16:52:09.763398477 +0000 UTC m=+0.018138206 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:52:10.120 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:09 smithi156 podman[90459]: 2024-04-24 16:52:09.876619288 +0000 UTC m=+0.131359008 container create d034252e58460df4305cc82e4980b62b93b85265a1890797d108720647983d04 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-24T16:52:10.120 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 podman[90459]: 2024-04-24 16:52:10.120960657 +0000 UTC m=+0.375700390 container init d034252e58460df4305cc82e4980b62b93b85265a1890797d108720647983d04 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-24T16:52:10.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:09 smithi070 ceph-mon[86496]: Health check failed: Degraded data redundancy: 9/627 objects degraded (1.435%), 4 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:10.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:09 smithi070 ceph-mon[81602]: Health check failed: Degraded data redundancy: 9/627 objects degraded (1.435%), 4 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:10.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 podman[90459]: 2024-04-24 16:52:10.124327841 +0000 UTC m=+0.379067566 container start d034252e58460df4305cc82e4980b62b93b85265a1890797d108720647983d04 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T16:52:10.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 podman[90459]: 2024-04-24 16:52:10.229434174 +0000 UTC m=+0.484173897 container attach d034252e58460df4305cc82e4980b62b93b85265a1890797d108720647983d04 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-24T16:52:10.910 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:52:10.910 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 bash[90459]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-24T16:52:10.911 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-24T16:52:10.912 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T16:52:10.912 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate[90474]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-24T16:52:10.912 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:10 smithi156 podman[90459]: 2024-04-24 16:52:10.638334021 +0000 UTC m=+0.893073741 container died d034252e58460df4305cc82e4980b62b93b85265a1890797d108720647983d04 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T16:52:11.241 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:10 smithi156 ceph-mon[65097]: pgmap v270: 161 pgs: 9 active+undersized, 4 active+undersized+degraded, 4 peering, 18 stale+active+clean, 126 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 132 op/s; 9/627 objects degraded (1.435%) 2024-04-24T16:52:11.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:10 smithi070 ceph-mon[86496]: pgmap v270: 161 pgs: 9 active+undersized, 4 active+undersized+degraded, 4 peering, 18 stale+active+clean, 126 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 132 op/s; 9/627 objects degraded (1.435%) 2024-04-24T16:52:11.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:10 smithi070 ceph-mon[81602]: pgmap v270: 161 pgs: 9 active+undersized, 4 active+undersized+degraded, 4 peering, 18 stale+active+clean, 126 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 132 op/s; 9/627 objects degraded (1.435%) 2024-04-24T16:52:11.557 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 podman[90596]: 2024-04-24 16:52:11.241684089 +0000 UTC m=+0.598665677 container remove d034252e58460df4305cc82e4980b62b93b85265a1890797d108720647983d04 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:52:11.557 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 podman[90662]: 2024-04-24 16:52:11.361854656 +0000 UTC m=+0.018800138 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:52:11.557 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 podman[90662]: 2024-04-24 16:52:11.459071211 +0000 UTC m=+0.116016692 container create e2d46d53eba41aefda13cc25602ec4a2a540e041ae973699664caa38a1931878 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T16:52:11.910 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 podman[90662]: 2024-04-24 16:52:11.644890195 +0000 UTC m=+0.301835677 container init e2d46d53eba41aefda13cc25602ec4a2a540e041ae973699664caa38a1931878 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:52:11.910 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 podman[90662]: 2024-04-24 16:52:11.647980943 +0000 UTC m=+0.304926433 container start e2d46d53eba41aefda13cc25602ec4a2a540e041ae973699664caa38a1931878 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:52:11.910 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 ceph-osd[90681]: -- 172.21.15.156:0/1031969434 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56106ed27680 con 0x56106ecf1400 2024-04-24T16:52:11.910 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 bash[90662]: e2d46d53eba41aefda13cc25602ec4a2a540e041ae973699664caa38a1931878 2024-04-24T16:52:11.910 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:11 smithi156 systemd[1]: Started Ceph osd.7 for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:52:12.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:52:12.196+0000 7f69cea227c0 -1 Falling back to public interface 2024-04-24T16:52:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[86496]: pgmap v271: 161 pgs: 37 active+undersized, 26 active+undersized+degraded, 4 peering, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 143 op/s; 81/627 objects degraded (12.919%) 2024-04-24T16:52:13.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:13.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:13.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:13.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[81602]: pgmap v271: 161 pgs: 37 active+undersized, 26 active+undersized+degraded, 4 peering, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 143 op/s; 81/627 objects degraded (12.919%) 2024-04-24T16:52:13.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:12 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:13.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:13.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:13.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:12 smithi156 ceph-mon[65097]: pgmap v271: 161 pgs: 37 active+undersized, 26 active+undersized+degraded, 4 peering, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 143 op/s; 81/627 objects degraded (12.919%) 2024-04-24T16:52:13.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:12 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:14.286 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:13 smithi156 ceph-mon[65097]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-24T16:52:14.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:13 smithi070 ceph-mon[86496]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-24T16:52:14.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:13 smithi070 ceph-mon[81602]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-24T16:52:15.176 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:14 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:15.177 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:14 smithi156 ceph-mon[65097]: pgmap v272: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 107 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:15.177 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:52:14.910+0000 7f69cea227c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-24T16:52:15.177 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:52:14.941+0000 7f69cea227c0 -1 osd.7 150 log_to_monitors true 2024-04-24T16:52:15.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:14 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:15.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:14 smithi070 ceph-mon[86496]: pgmap v272: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 107 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:15.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:14 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:15.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:14 smithi070 ceph-mon[81602]: pgmap v272: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 107 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:16.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:15 smithi156 ceph-mon[65097]: from='osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:52:16.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:15 smithi156 ceph-mon[65097]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:52:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:15 smithi070 ceph-mon[81602]: from='osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:52:16.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:15 smithi070 ceph-mon[81602]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:52:16.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:15 smithi070 ceph-mon[86496]: from='osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:52:16.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:15 smithi070 ceph-mon[86496]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T16:52:16.549 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:52:16.234+0000 7f69bbe72700 -1 osd.7 150 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T16:52:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:52:16] "GET /metrics HTTP/1.1" 200 38288 "" "Prometheus/2.43.0" 2024-04-24T16:52:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: pgmap v273: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 71 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:17.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: from='osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: osdmap e153: 8 total, 7 up, 8 in 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[86496]: from='osd.7 ' entity='osd.7' 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: pgmap v273: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 71 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: from='osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: osdmap e153: 8 total, 7 up, 8 in 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:52:17.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:17.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:17.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:16 smithi070 ceph-mon[81602]: from='osd.7 ' entity='osd.7' 2024-04-24T16:52:17.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: pgmap v273: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 71 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:17.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T16:52:17.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: from='osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:52:17.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: osdmap e153: 8 total, 7 up, 8 in 2024-04-24T16:52:17.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-04-24T16:52:17.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:17.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:17.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:16 smithi156 ceph-mon[65097]: from='osd.7 ' entity='osd.7' 2024-04-24T16:52:18.132 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.132 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:18.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 85/627 objects degraded (13.557%), 28 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:18.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:52:18.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176] boot 2024-04-24T16:52:18.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: osdmap e154: 8 total, 8 up, 8 in 2024-04-24T16:52:18.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:18 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:52:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 85/627 objects degraded (13.557%), 28 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:18.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176] boot 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: osdmap e154: 8 total, 8 up, 8 in 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 85/627 objects degraded (13.557%), 28 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:18.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:18.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-24T16:52:18.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: osd.7 [v2:172.21.15.156:6824/2350467176,v1:172.21.15.156:6825/2350467176] boot 2024-04-24T16:52:18.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: osdmap e154: 8 total, 8 up, 8 in 2024-04-24T16:52:18.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:18 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T16:52:19.355 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:19 smithi070 ceph-mon[86496]: pgmap v276: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:19.355 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:19 smithi070 ceph-mon[86496]: osdmap e155: 8 total, 8 up, 8 in 2024-04-24T16:52:19.356 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:19 smithi070 ceph-mon[81602]: pgmap v276: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:19.356 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:19 smithi070 ceph-mon[81602]: osdmap e155: 8 total, 8 up, 8 in 2024-04-24T16:52:19.356 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:52:19.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:19 smithi156 ceph-mon[65097]: pgmap v276: 161 pgs: 39 active+undersized, 28 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 85/627 objects degraded (13.557%) 2024-04-24T16:52:19.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:19 smithi156 ceph-mon[65097]: osdmap e155: 8 total, 8 up, 8 in 2024-04-24T16:52:20.008 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (8m) 2m ago 31m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (8m) 3s ago 30m 80.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 2m ago 30m 64.0M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (10m) 3s ago 35m 454M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (20m) 2m ago 36m 509M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (6m) 2m ago 37m 51.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (7m) 3s ago 35m 50.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (6m) 2m ago 35m 44.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (9m) 2m ago 31m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (9m) 3s ago 31m 15.6M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (4m) 2m ago 35m 52.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:52:20.009 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (4m) 2m ago 34m 54.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (3m) 2m ago 34m 44.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (2m) 2m ago 33m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (113s) 3s ago 33m 55.9M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (78s) 3s ago 33m 48.3M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (43s) 3s ago 32m 47.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (8s) 3s ago 32m 11.6M 3248M 19.0.0-3285-g5904d294 8bb2087275ff e2d46d53eba4 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (9m) 3s ago 31m 69.3M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (30m) 2m ago 30m 78.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:52:20.010 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (30m) 3s ago 30m 78.7M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:52:20.411 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:52:20.411 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:52:20.412 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:52:20.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:52:20.412 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:52:20.412 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:52:20.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:52:20.412 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:52:20.412 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 8 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 13 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:52:20.413 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:52:20.760 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:52:20.760 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:52:20.760 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:52:20.760 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-24T16:52:20.760 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-04-24T16:52:20.760 INFO:teuthology.orchestra.run.smithi070.stdout: "osd" 2024-04-24T16:52:20.760 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:52:20.761 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "8/8 daemons upgraded", 2024-04-24T16:52:20.761 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading osd daemons", 2024-04-24T16:52:20.761 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:52:20.761 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:52:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='client.64540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: pgmap v278: 161 pgs: 10 peering, 32 active+undersized, 25 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-04-24T16:52:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1363777961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-24T16:52:21.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-24T16:52:21.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='client.64540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: pgmap v278: 161 pgs: 10 peering, 32 active+undersized, 25 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1363777961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-24T16:52:21.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-24T16:52:21.546 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='client.64540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: pgmap v278: 161 pgs: 10 peering, 32 active+undersized, 25 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1363777961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-24T16:52:21.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-24T16:52:21.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-24T16:52:21.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-24T16:52:21.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-24T16:52:21.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-24T16:52:21.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: pgmap v279: 161 pgs: 10 peering, 17 active+undersized, 15 active+undersized+degraded, 119 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%) 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: pgmap v280: 161 pgs: 10 peering, 17 active+undersized, 15 active+undersized+degraded, 119 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%) 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all osd 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: Upgrade: Setting require_osd_release to 19 squid 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: osdmap e156: 8 total, 8 up, 8 in 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:52:22.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:22.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:22 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: pgmap v279: 161 pgs: 10 peering, 17 active+undersized, 15 active+undersized+degraded, 119 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%) 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: pgmap v280: 161 pgs: 10 peering, 17 active+undersized, 15 active+undersized+degraded, 119 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%) 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all osd 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: Upgrade: Setting require_osd_release to 19 squid 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: osdmap e156: 8 total, 8 up, 8 in 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:52:22.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: pgmap v279: 161 pgs: 10 peering, 17 active+undersized, 15 active+undersized+degraded, 119 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%) 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: pgmap v280: 161 pgs: 10 peering, 17 active+undersized, 15 active+undersized+degraded, 119 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%) 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all osd 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: Upgrade: Setting require_osd_release to 19 squid 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: osdmap e156: 8 total, 8 up, 8 in 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.546 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:52:22.547 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:52:22.548 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-24T16:52:22.549 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.550 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:22.551 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:52:22.552 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:22 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mds 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nfs 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Upgrade: Finalizing container_image settings 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Upgrade: Complete! 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: pgmap v282: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: pgmap v283: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: pgmap v284: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: pgmap v285: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.331 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 15 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:23.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:23 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mds 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nfs 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Upgrade: Finalizing container_image settings 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Upgrade: Complete! 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: pgmap v282: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: pgmap v283: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: pgmap v284: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: pgmap v285: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.340 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 15 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mds 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nfs 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Upgrade: Finalizing container_image settings 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Upgrade: Complete! 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: pgmap v282: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: pgmap v283: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: pgmap v284: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: pgmap v285: 161 pgs: 10 peering, 1 active+undersized, 2 active+undersized+degraded, 148 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 4/627 objects degraded (0.638%) 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 15 pgs degraded (PG_DEGRADED) 2024-04-24T16:52:23.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:23 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:24.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:24 smithi156 ceph-mon[65097]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/627 objects degraded (0.638%), 2 pgs degraded) 2024-04-24T16:52:24.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:24 smithi070 ceph-mon[86496]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/627 objects degraded (0.638%), 2 pgs degraded) 2024-04-24T16:52:24.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:24 smithi070 ceph-mon[81602]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/627 objects degraded (0.638%), 2 pgs degraded) 2024-04-24T16:52:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:25 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:25 smithi156 ceph-mon[65097]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:52:25.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:25 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:25.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:25 smithi070 ceph-mon[81602]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:52:25.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:25 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:25.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:25 smithi070 ceph-mon[86496]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:52:26.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:52:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:52:26] "GET /metrics HTTP/1.1" 200 38371 "" "Prometheus/2.43.0" 2024-04-24T16:52:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:27 smithi156 ceph-mon[65097]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:27.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:27 smithi070 ceph-mon[81602]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:27.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:27 smithi070 ceph-mon[86496]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:29.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:29 smithi156 ceph-mon[65097]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:52:29.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:29 smithi070 ceph-mon[86496]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:52:29.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:29 smithi070 ceph-mon[81602]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-04-24T16:52:31.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:31 smithi156 ceph-mon[65097]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-04-24T16:52:31.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:31 smithi070 ceph-mon[86496]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-04-24T16:52:31.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:31 smithi070 ceph-mon[81602]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-04-24T16:52:33.344 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[86496]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:52:33.344 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:33.344 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:33.345 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:33.345 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[81602]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:52:33.345 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:33.345 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:33.345 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:33 smithi156 ceph-mon[65097]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:52:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:52:35.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:35 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:35.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:35 smithi156 ceph-mon[65097]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:35.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:35 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:35.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:35 smithi070 ceph-mon[86496]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:35.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:35 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:35.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:35 smithi070 ceph-mon[81602]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:52:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:52:36] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-04-24T16:52:37.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:37 smithi156 ceph-mon[65097]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:37.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:37 smithi070 ceph-mon[81602]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:37.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:37 smithi070 ceph-mon[86496]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:39.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:39 smithi156 ceph-mon[65097]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:39.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:39 smithi070 ceph-mon[81602]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:39.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:39 smithi070 ceph-mon[86496]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:41.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:41 smithi156 ceph-mon[65097]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:41.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:41 smithi070 ceph-mon[86496]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:41.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:41 smithi070 ceph-mon[81602]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:43.359 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:43 smithi070 ceph-mon[86496]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:43.359 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:43 smithi070 ceph-mon[81602]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:43.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:43 smithi156 ceph-mon[65097]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:45.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:45 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:45.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:45 smithi156 ceph-mon[65097]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:45.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:45 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:45.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:45 smithi070 ceph-mon[81602]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:45.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:45 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:45.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:45 smithi070 ceph-mon[86496]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-24T16:52:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:52:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:52:46] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-04-24T16:52:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:47 smithi156 ceph-mon[65097]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:47.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:47 smithi070 ceph-mon[86496]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:47.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:47 smithi070 ceph-mon[81602]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:47.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:52:49.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:49 smithi156 ceph-mon[65097]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:49.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:49 smithi070 ceph-mon[86496]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:49.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:49 smithi070 ceph-mon[81602]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:51.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:51 smithi156 ceph-mon[65097]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:51.512 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:51 smithi070 ceph-mon[86496]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:51.512 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:51 smithi070 ceph-mon[81602]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:51.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:52:52.244 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:52 smithi070 ceph-mon[86496]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:52.245 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:52 smithi070 ceph-mon[81602]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:52 smithi156 ceph-mon[65097]: from='client.54627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:53.364 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:53 smithi070 ceph-mon[86496]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:53.364 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:53 smithi070 ceph-mon[81602]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:53.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:53 smithi156 ceph-mon[65097]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (9m) 2m ago 31m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (9m) 37s ago 31m 80.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 2m ago 30m 64.0M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (10m) 37s ago 35m 454M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (20m) 2m ago 37m 509M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (7m) 2m ago 37m 51.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (8m) 37s ago 36m 50.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (7m) 2m ago 36m 44.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:52:53.944 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (10m) 2m ago 32m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (10m) 37s ago 32m 15.6M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (5m) 2m ago 35m 52.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (4m) 2m ago 35m 54.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (3m) 2m ago 34m 44.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (3m) 2m ago 34m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (2m) 37s ago 34m 55.9M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (112s) 37s ago 33m 48.3M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (77s) 37s ago 33m 47.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (42s) 37s ago 32m 11.6M 3248M 19.0.0-3285-g5904d294 8bb2087275ff e2d46d53eba4 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (9m) 37s ago 32m 69.3M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (30m) 2m ago 30m 78.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:52:53.945 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (30m) 37s ago 30m 78.7M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:52:54.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-04-24T16:52:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:55 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:55 smithi156 ceph-mon[65097]: from='client.64570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:55 smithi156 ceph-mon[65097]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:55.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:55 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:55.520 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:55 smithi070 ceph-mon[86496]: from='client.64570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:55.521 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:55 smithi070 ceph-mon[86496]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:55.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:55 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:52:55.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:55 smithi070 ceph-mon[81602]: from='client.64570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:52:55.521 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:55 smithi070 ceph-mon[81602]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:56.655 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:52:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:52:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:52:56] "GET /metrics HTTP/1.1" 200 38379 "" "Prometheus/2.43.0" 2024-04-24T16:52:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:57 smithi070 ceph-mon[81602]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:57.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:57 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1511158779' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:57 smithi070 ceph-mon[86496]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:57.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:57 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1511158779' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:57.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-04-24T16:52:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:57 smithi156 ceph-mon[65097]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:52:57.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:57 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1511158779' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:52:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:52:59 smithi070 ceph-mon[86496]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:52:59 smithi070 ceph-mon[81602]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)" 2024-04-24T16:52:59.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:52:59 smithi156 ceph-mon[65097]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:52:59.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:53:00.285 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:00 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3653196433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:00.286 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:00 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3653196433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:00 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3653196433' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:01.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:01 smithi070 ceph-mon[86496]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:01.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:01 smithi070 ceph-mon[81602]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:01.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:01 smithi156 ceph-mon[65097]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:02.077 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:53:02.077 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": null, 2024-04-24T16:53:02.077 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": false, 2024-04-24T16:53:02.077 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "", 2024-04-24T16:53:02.077 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:53:02.077 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": null, 2024-04-24T16:53:02.077 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-04-24T16:53:02.078 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:53:02.078 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:53:02.377 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:02.378 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:02.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:02.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:53:03.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:03 smithi070 ceph-mon[86496]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:03.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:03 smithi070 ceph-mon[86496]: from='client.54651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:03.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:03 smithi070 ceph-mon[81602]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:03.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:03 smithi070 ceph-mon[81602]: from='client.54651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:03 smithi156 ceph-mon[65097]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:03.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:03 smithi156 ceph-mon[65097]: from='client.54651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:04.750 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:53:04.750 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:53:04.750 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:53:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:05 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:05 smithi070 ceph-mon[81602]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:05.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:05 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2403705069' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:53:05.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:05 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:05.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:05 smithi070 ceph-mon[86496]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:05.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:05 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2403705069' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:53:05.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-04-24T16:53:05.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:05 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:05.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:05 smithi156 ceph-mon[65097]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:05.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:05 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2403705069' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:53:06.601 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:53:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:53:06] "GET /metrics HTTP/1.1" 200 38378 "" "Prometheus/2.43.0" 2024-04-24T16:53:07.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:07 smithi070 ceph-mon[81602]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:07.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:07 smithi070 ceph-mon[86496]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:07.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:07 smithi156 ceph-mon[65097]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:09 smithi156 ceph-mon[65097]: from='client.64600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:09.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:09 smithi156 ceph-mon[65097]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:09.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:09 smithi070 ceph-mon[86496]: from='client.64600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:09.415 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:09 smithi070 ceph-mon[86496]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:09.415 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:09 smithi070 ceph-mon[81602]: from='client.64600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:09.415 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:09 smithi070 ceph-mon[81602]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:09.461 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:53:10.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-24T16:53:10.630 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[86496]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:10.631 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:10.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:10.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:10.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-04-24T16:53:10.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:10.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:10.632 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:10 smithi070 ceph-mon[81602]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:53:10.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:10.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:10 smithi156 ceph-mon[65097]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:53:12.250 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:53:12.731 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:12 smithi070 ceph-mon[86496]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:12.731 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:12 smithi070 ceph-mon[81602]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:12.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:12 smithi156 ceph-mon[65097]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:12.933 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:53:12.933 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (9m) 3m ago 31m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:53:12.933 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (9m) 56s ago 31m 80.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:53:12.933 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 3m ago 31m 64.0M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:53:12.933 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (11m) 56s ago 36m 454M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:53:12.933 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (21m) 3m ago 37m 509M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:53:12.933 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (7m) 3m ago 37m 51.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (8m) 56s ago 36m 50.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (7m) 3m ago 36m 44.2M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (10m) 3m ago 32m 16.5M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (10m) 56s ago 32m 15.6M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (5m) 3m ago 35m 52.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (5m) 3m ago 35m 54.6M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (4m) 3m ago 35m 44.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (3m) 3m ago 34m 12.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (2m) 56s ago 34m 55.9M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (2m) 56s ago 34m 48.3M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (96s) 56s ago 33m 47.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (61s) 56s ago 33m 11.6M 3248M 19.0.0-3285-g5904d294 8bb2087275ff e2d46d53eba4 2024-04-24T16:53:12.934 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (10m) 56s ago 32m 69.3M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:53:12.935 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (31m) 3m ago 31m 78.0M - 17.2.0 e1d6a67b021e 3a874448de3c 2024-04-24T16:53:12.935 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (31m) 56s ago 31m 78.7M - 17.2.0 e1d6a67b021e b934d5c4436c 2024-04-24T16:53:13.355 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:53:13.355 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:53:13.355 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:53:13.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:53:13.356 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:53:13.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:53:13.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:53:13.356 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:53:13.356 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 8 2024-04-24T16:53:13.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:53:13.357 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:53:13.357 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-24T16:53:13.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:53:13.357 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:53:13.357 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-24T16:53:13.357 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 13 2024-04-24T16:53:13.358 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:53:13.358 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:53:13.709 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:53:13.709 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:53:13.709 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:53:13.709 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-04-24T16:53:13.710 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:53:13.710 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "0/2 daemons upgraded", 2024-04-24T16:53:13.710 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading rgw daemons", 2024-04-24T16:53:13.710 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:53:13.710 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='client.64606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='client.54681 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mgr 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mon 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all crash 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all osd 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mds 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1110735051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='client.64606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='client.54681 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mgr 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mon 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all crash 2024-04-24T16:53:14.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all osd 2024-04-24T16:53:14.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mds 2024-04-24T16:53:14.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:14 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1110735051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='client.64606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='client.54681 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mgr 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mon 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all crash 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all osd 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mds 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:14.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:14 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1110735051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:15.322 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[86496]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[86496]: from='client.54693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[81602]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[81602]: from='client.54693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:53:15.323 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:15 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:15 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:15 smithi156 ceph-mon[65097]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:15 smithi156 ceph-mon[65097]: from='client.54693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:15.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi070.vhdldq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:53:15.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:15 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:16.334 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:16 smithi070 ceph-mon[86496]: Upgrade: Updating rgw.foo.smithi070.vhdldq (1/2) 2024-04-24T16:53:16.335 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:16 smithi070 ceph-mon[86496]: Deploying daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:53:16.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:16 smithi070 ceph-mon[81602]: Upgrade: Updating rgw.foo.smithi070.vhdldq (1/2) 2024-04-24T16:53:16.335 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:16 smithi070 ceph-mon[81602]: Deploying daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:53:16.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:16 smithi156 ceph-mon[65097]: Upgrade: Updating rgw.foo.smithi070.vhdldq (1/2) 2024-04-24T16:53:16.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:16 smithi156 ceph-mon[65097]: Deploying daemon rgw.foo.smithi070.vhdldq on smithi070 2024-04-24T16:53:16.618 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:53:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:53:16] "GET /metrics HTTP/1.1" 200 38378 "" "Prometheus/2.43.0" 2024-04-24T16:53:17.252 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:17 smithi070 ceph-mon[81602]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:17.252 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:17.252 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:17 smithi070 ceph-mon[86496]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:17.252 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:17.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:17 smithi156 ceph-mon[65097]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:17.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:19.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:19 smithi156 ceph-mon[65097]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:19.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:19.385 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:19.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:19 smithi070 ceph-mon[86496]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:19.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:19.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:19.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:19 smithi070 ceph-mon[81602]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:53:19.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:19.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:21.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:21 smithi156 ceph-mon[65097]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:53:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:53:21.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:21 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:21.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[86496]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:53:21.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:21.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:53:21.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[81602]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:53:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi156.pmtzwp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-24T16:53:21.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:21 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:22.300 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:22 smithi156 ceph-mon[65097]: Upgrade: Updating rgw.foo.smithi156.pmtzwp (2/2) 2024-04-24T16:53:22.300 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:22 smithi156 ceph-mon[65097]: Deploying daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:53:22.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:22 smithi070 ceph-mon[86496]: Upgrade: Updating rgw.foo.smithi156.pmtzwp (2/2) 2024-04-24T16:53:22.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:22 smithi070 ceph-mon[86496]: Deploying daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:53:22.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:22 smithi070 ceph-mon[81602]: Upgrade: Updating rgw.foo.smithi156.pmtzwp (2/2) 2024-04-24T16:53:22.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:22 smithi070 ceph-mon[81602]: Deploying daemon rgw.foo.smithi156.pmtzwp on smithi156 2024-04-24T16:53:23.388 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:23 smithi070 ceph-mon[86496]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 59 op/s 2024-04-24T16:53:23.389 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:23 smithi070 ceph-mon[81602]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 59 op/s 2024-04-24T16:53:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:23 smithi156 ceph-mon[65097]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 59 op/s 2024-04-24T16:53:25.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:25.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[86496]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-24T16:53:25.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:25.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:25.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:25.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:25.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[81602]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-24T16:53:25.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:25.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:25.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:25 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:25.307 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:25 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:25.307 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:25 smithi156 ceph-mon[65097]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-24T16:53:25.307 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:25 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:25.307 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:25 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:25.307 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:25 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:26.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:53:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:53:26] "GET /metrics HTTP/1.1" 200 38378 "" "Prometheus/2.43.0" 2024-04-24T16:53:27.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:27 smithi156 ceph-mon[65097]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 85 B/s wr, 114 op/s 2024-04-24T16:53:27.277 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:27 smithi070 ceph-mon[86496]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 85 B/s wr, 114 op/s 2024-04-24T16:53:27.277 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:27 smithi070 ceph-mon[81602]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 85 B/s wr, 114 op/s 2024-04-24T16:53:29.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[86496]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 170 B/s wr, 155 op/s 2024-04-24T16:53:29.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[81602]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 170 B/s wr, 155 op/s 2024-04-24T16:53:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:28 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:28 smithi156 ceph-mon[65097]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 170 B/s wr, 155 op/s 2024-04-24T16:53:29.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:28 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:29 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:29 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:29 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:29.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:29 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:30.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:29 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:31.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:30 smithi156 ceph-mon[65097]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-04-24T16:53:31.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:30 smithi070 ceph-mon[81602]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-04-24T16:53:31.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:30 smithi070 ceph-mon[86496]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-04-24T16:53:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-24T16:53:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi070.vhdldq"}]: dispatch 2024-04-24T16:53:33.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi070.vhdldq"}]': finished 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi156.pmtzwp"}]: dispatch 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi156.pmtzwp"}]': finished 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-24T16:53:33.296 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi070.vhdldq"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi070.vhdldq"}]': finished 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi156.pmtzwp"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi156.pmtzwp"}]': finished 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.298 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.299 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-24T16:53:33.300 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.301 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:53:33.302 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:53:33.303 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.304 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.305 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.306 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:33 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi070.vhdldq"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi070.vhdldq"}]': finished 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi156.pmtzwp"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi156.pmtzwp"}]': finished 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.412 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.413 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-24T16:53:33.414 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-24T16:53:33.415 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:33.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:53:33.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:53:33.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:53:33.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:33 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:34.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 187 B/s wr, 120 op/s 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 115 B/s wr, 113 op/s 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all rgw 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nfs 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Upgrade: Finalizing container_image settings 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Upgrade: Complete! 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 143 B/s wr, 91 op/s 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: Checking dashboard <-> RGW credentials 2024-04-24T16:53:34.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[86496]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 187 B/s wr, 120 op/s 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 115 B/s wr, 113 op/s 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all rgw 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nfs 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Upgrade: Finalizing container_image settings 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Upgrade: Complete! 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 143 B/s wr, 91 op/s 2024-04-24T16:53:34.295 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-04-24T16:53:34.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: Checking dashboard <-> RGW credentials 2024-04-24T16:53:34.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:53:34.296 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:34 smithi070 ceph-mon[81602]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:53:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 187 B/s wr, 120 op/s 2024-04-24T16:53:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 115 B/s wr, 113 op/s 2024-04-24T16:53:34.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all rgw 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nfs 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all nvmeof 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Upgrade: Finalizing container_image settings 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Upgrade: Complete! 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 143 B/s wr, 91 op/s 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: Checking dashboard <-> RGW credentials 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-24T16:53:34.411 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:34 smithi156 ceph-mon[65097]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:53:35.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:35 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:35.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:35 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:35.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:35 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:36.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:36 smithi070 ceph-mon[86496]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:53:36.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:36 smithi070 ceph-mon[81602]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:53:36.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:36 smithi156 ceph-mon[65097]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:53:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:53:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:53:36] "GET /metrics HTTP/1.1" 200 38409 "" "Prometheus/2.43.0" 2024-04-24T16:53:38.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:38 smithi070 ceph-mon[81602]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-04-24T16:53:38.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:38 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:38.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:38 smithi070 ceph-mon[86496]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-04-24T16:53:38.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:38 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:38 smithi156 ceph-mon[65097]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-04-24T16:53:38.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:38 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:53:40.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:40 smithi070 ceph-mon[86496]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-04-24T16:53:40.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:40 smithi070 ceph-mon[81602]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-04-24T16:53:40.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:40 smithi156 ceph-mon[65097]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-04-24T16:53:42.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:42 smithi070 ceph-mon[86496]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 4 op/s 2024-04-24T16:53:42.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:42 smithi070 ceph-mon[81602]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 4 op/s 2024-04-24T16:53:42.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:42 smithi156 ceph-mon[65097]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 4 op/s 2024-04-24T16:53:44.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:44 smithi070 ceph-mon[86496]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-24T16:53:44.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:44 smithi070 ceph-mon[81602]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-24T16:53:44.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:44 smithi156 ceph-mon[65097]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-24T16:53:44.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:53:45.026 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:45 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:45.297 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:45 smithi070 ceph-mon[86496]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:45.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:45 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:45.297 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:45 smithi070 ceph-mon[81602]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:45.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:45 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:45.416 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:45 smithi156 ceph-mon[65097]: from='client.54771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:46.122 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:46 smithi070 ceph-mon[86496]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-24T16:53:46.122 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:46 smithi070 ceph-mon[81602]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-24T16:53:46.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:46 smithi156 ceph-mon[65097]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-24T16:53:46.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:53:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:53:46] "GET /metrics HTTP/1.1" 200 38409 "" "Prometheus/2.43.0" 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (10m) 19s ago 32m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (10m) 18s ago 32m 80.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 19s ago 31m 66.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (11m) 18s ago 36m 455M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (21m) 19s ago 38m 515M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (8m) 19s ago 38m 58.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (9m) 18s ago 37m 53.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (7m) 19s ago 37m 50.8M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:53:46.802 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (11m) 19s ago 33m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (11m) 18s ago 33m 15.4M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (6m) 19s ago 36m 61.4M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (5m) 19s ago 36m 64.1M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (4m) 19s ago 35m 51.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (3m) 19s ago 35m 59.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (3m) 18s ago 34m 60.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (2m) 18s ago 34m 50.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (2m) 18s ago 34m 50.2M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (95s) 18s ago 33m 54.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff e2d46d53eba4 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (10m) 18s ago 33m 75.1M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (28s) 19s ago 31m 74.8M - 19.0.0-3285-g5904d294 8bb2087275ff 8aa1086a1f79 2024-04-24T16:53:46.803 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (23s) 18s ago 31m 74.6M - 19.0.0-3285-g5904d294 8bb2087275ff f9d108c6acab 2024-04-24T16:53:46.910 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:53:46 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=cleanup t=2024-04-24T16:53:46.549862418Z level=info msg="Completed cleanup jobs" duration=3.368441ms 2024-04-24T16:53:47.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-04-24T16:53:48.099 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:48 smithi070 ceph-mon[86496]: from='client.54777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:48.100 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:48 smithi070 ceph-mon[86496]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-24T16:53:48.100 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:48 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:48.100 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:48 smithi070 ceph-mon[81602]: from='client.54777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:48.100 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:48 smithi070 ceph-mon[81602]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-24T16:53:48.100 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:48 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:48.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:48 smithi156 ceph-mon[65097]: from='client.54777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:48.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:48 smithi156 ceph-mon[65097]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-24T16:53:48.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:48 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:53:49.587 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:53:50.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:50 smithi070 ceph-mon[86496]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-24T16:53:50.245 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:50 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3997208895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:50.245 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:50 smithi070 ceph-mon[81602]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-24T16:53:50.245 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:50 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3997208895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:50.268 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-04-24T16:53:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:50 smithi156 ceph-mon[65097]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-24T16:53:50.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:50 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3997208895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:52.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:52 smithi070 ceph-mon[81602]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:52.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:52 smithi070 ceph-mon[86496]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:52.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:52 smithi156 ceph-mon[65097]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:52.417 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)" 2024-04-24T16:53:53.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:53:53.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:53 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3762898927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:53.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:53 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3762898927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:53.311 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:53 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3762898927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:53:54.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:54 smithi070 ceph-mon[86496]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:54.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:54 smithi070 ceph-mon[81602]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:54.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:54 smithi156 ceph-mon[65097]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:55.075 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:53:55.075 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": null, 2024-04-24T16:53:55.076 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": false, 2024-04-24T16:53:55.076 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "", 2024-04-24T16:53:55.076 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [], 2024-04-24T16:53:55.076 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": null, 2024-04-24T16:53:55.076 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "", 2024-04-24T16:53:55.076 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:53:55.076 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:53:55.382 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:55 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:55.382 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:55 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:55.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:55 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:53:55.691 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:53:56.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:56 smithi070 ceph-mon[86496]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:56.294 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:56 smithi070 ceph-mon[86496]: from='client.54795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:56.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:56 smithi070 ceph-mon[81602]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:56.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:56 smithi070 ceph-mon[81602]: from='client.54795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:56 smithi156 ceph-mon[65097]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:53:56.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:56 smithi156 ceph-mon[65097]: from='client.54795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:53:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:53:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:53:56] "GET /metrics HTTP/1.1" 200 38407 "" "Prometheus/2.43.0" 2024-04-24T16:53:57.697 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:53:57.697 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:53:57.697 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:53:58.288 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:58 smithi070 ceph-mon[86496]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:58.289 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:53:58 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3237297422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:53:58.289 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:58 smithi070 ceph-mon[81602]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:58.289 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:53:58 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3237297422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:53:58.311 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-24T16:53:58.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:58 smithi156 ceph-mon[65097]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:53:58.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:53:58 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3237297422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:54:00.230 INFO:teuthology.orchestra.run.smithi070.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:54:00.243 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:00 smithi070 ceph-mon[86496]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:00.243 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:00 smithi070 ceph-mon[81602]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:00.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:00 smithi156 ceph-mon[65097]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:00.951 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T16:54:00.962 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-04-24T16:54:00.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-24T16:54:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: from='client.54807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:54:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[86496]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: from='client.54807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:54:01.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:54:01.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:01.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:01.545 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:01 smithi070 ceph-mon[81602]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:54:01.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: from='client.54807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:01.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:54:01.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:01.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:01.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:01.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:54:01.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:54:01.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:01.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:01.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:01 smithi156 ceph-mon[65097]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T16:54:02.480 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:02.480 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:03.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:03 smithi070 ceph-mon[81602]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:03.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:03 smithi070 ceph-mon[86496]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:03.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:03 smithi156 ceph-mon[65097]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:04.551 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:54:04.720 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mgr 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mon 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all crash 2024-04-24T16:54:04.721 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all osd 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all mds 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all rgw 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.722 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.723 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mgr 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mon 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all crash 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all osd 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all mds 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.724 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all rgw 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:54:04.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:04 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Target is version 19.0.0-3285-g5904d294 (squid) 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a'] 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mgr 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mon 2024-04-24T16:54:04.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all crash 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all osd 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all mds 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all rgw 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.912 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all rbd-mirror 2024-04-24T16:54:04.913 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:04.913 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.913 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-24T16:54:04.913 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:04.913 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: Upgrade: Setting container_image for all ceph-exporter 2024-04-24T16:54:04.913 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:04 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:05.241 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (10m) 37s ago 32m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (10m) 37s ago 32m 80.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 37s ago 31m 66.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (11m) 37s ago 37m 455M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (22m) 37s ago 38m 515M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (8m) 37s ago 38m 58.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (9m) 37s ago 37m 53.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (8m) 37s ago 37m 50.8M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:54:05.242 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (11m) 37s ago 33m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:54:05.243 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (11m) 37s ago 33m 15.4M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:54:05.243 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (6m) 37s ago 36m 61.4M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:54:05.243 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (6m) 37s ago 36m 64.1M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:54:05.243 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (5m) 37s ago 36m 51.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:54:05.243 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (4m) 37s ago 35m 59.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:54:05.243 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (3m) 37s ago 35m 60.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:54:05.243 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (3m) 37s ago 34m 50.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:54:05.244 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (2m) 37s ago 34m 50.2M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:54:05.244 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (113s) 37s ago 34m 54.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff e2d46d53eba4 2024-04-24T16:54:05.244 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (11m) 37s ago 33m 75.1M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:54:05.244 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (47s) 37s ago 32m 74.8M - 19.0.0-3285-g5904d294 8bb2087275ff 8aa1086a1f79 2024-04-24T16:54:05.244 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (41s) 37s ago 32m 74.6M - 19.0.0-3285-g5904d294 8bb2087275ff f9d108c6acab 2024-04-24T16:54:05.666 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:54:05.666 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:54:05.666 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:54:05.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:05.666 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:54:05.666 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 8 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 15 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:54:05.667 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:54:05.766 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:05 smithi070 ceph-mon[86496]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:05.766 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:05 smithi070 ceph-mon[86496]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:05.766 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:05 smithi070 ceph-mon[86496]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:05.766 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:05 smithi070 ceph-mon[81602]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:05.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:05 smithi070 ceph-mon[81602]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:05.767 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:05 smithi070 ceph-mon[81602]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:05.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:05 smithi156 ceph-mon[65097]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:05.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:05 smithi156 ceph-mon[65097]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:05.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:05 smithi156 ceph-mon[65097]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:06.015 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:54:06.015 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:54:06.015 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:54:06.015 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "osd", 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw", 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "mon", 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "15/21 daemons upgraded", 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Currently upgrading iscsi daemons", 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": false 2024-04-24T16:54:06.016 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:54:06.440 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:54:06.440 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:54:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:54:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:54:06] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: Upgrade: Updating iscsi.foo.smithi070.gcoqkr 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: Deploying daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1945836446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: from='client.54837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3439407680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:54:06.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: Upgrade: Updating iscsi.foo.smithi070.gcoqkr 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: Deploying daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1945836446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: from='client.54837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:06.795 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:06 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3439407680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:54:06.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:06.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: Upgrade: Updating iscsi.foo.smithi070.gcoqkr 2024-04-24T16:54:06.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:06.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi070.gcoqkr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-24T16:54:06.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:06.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: Deploying daemon iscsi.foo.smithi070.gcoqkr on smithi070 2024-04-24T16:54:06.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1945836446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:06.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: from='client.54837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:06.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:06 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3439407680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:54:07.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:07 smithi156 ceph-mon[65097]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:08.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:07 smithi070 ceph-mon[86496]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:08.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:07 smithi070 ceph-mon[81602]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:09.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:09 smithi156 ceph-mon[65097]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:10.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:09 smithi070 ceph-mon[81602]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:10.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:09 smithi070 ceph-mon[86496]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:54:11.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:11 smithi156 ceph-mon[65097]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:54:12.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:11 smithi070 ceph-mon[86496]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:54:12.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:11 smithi070 ceph-mon[81602]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:54:13.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:13 smithi070 ceph-mon[86496]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:13.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:13 smithi070 ceph-mon[81602]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:13.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:13 smithi156 ceph-mon[65097]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:14.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:14 smithi156 ceph-mon[65097]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:15.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:14 smithi070 ceph-mon[86496]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:15.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:14 smithi070 ceph-mon[81602]: from='client.15444 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:15.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:15 smithi156 ceph-mon[65097]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:16.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:15 smithi070 ceph-mon[86496]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:16.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:15 smithi070 ceph-mon[81602]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:54:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:54:16] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-04-24T16:54:17.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:17 smithi156 ceph-mon[65097]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:17.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:18.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:17 smithi070 ceph-mon[86496]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:18.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:17 smithi070 ceph-mon[81602]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:18.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:54:18 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:54:18.623+0000 7f224bc90700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. 2024-04-24T16:54:19.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 0 op/s 2024-04-24T16:54:19.911 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:20.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 0 op/s 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. 2024-04-24T16:54:20.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:20.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-24T16:54:20.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:54:20.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:54:20.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:54:20.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-04-24T16:54:20.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 804 B/s rd, 0 op/s 2024-04-24T16:54:20.045 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:20.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:20 smithi156 ceph-mon[65097]: Health check failed: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-24T16:54:21.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:20 smithi070 ceph-mon[86496]: Health check failed: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-24T16:54:21.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:20 smithi070 ceph-mon[81602]: Health check failed: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-24T16:54:21.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:21 smithi156 ceph-mon[65097]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:54:22.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:21 smithi070 ceph-mon[86496]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:54:22.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:21 smithi070 ceph-mon[81602]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-24T16:54:23.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:23 smithi156 ceph-mon[65097]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-24T16:54:24.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:23 smithi070 ceph-mon[86496]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-24T16:54:24.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:23 smithi070 ceph-mon[81602]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-24T16:54:25.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:25 smithi156 ceph-mon[65097]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-24T16:54:26.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:25 smithi070 ceph-mon[86496]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-24T16:54:26.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:25 smithi070 ceph-mon[81602]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-24T16:54:26.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:54:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:54:26] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-04-24T16:54:27.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:27 smithi156 ceph-mon[65097]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-24T16:54:28.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:27 smithi070 ceph-mon[86496]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-24T16:54:28.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:27 smithi070 ceph-mon[81602]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-04-24T16:54:29.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:29 smithi156 ceph-mon[65097]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-04-24T16:54:30.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:29 smithi070 ceph-mon[81602]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-04-24T16:54:30.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:29 smithi070 ceph-mon[86496]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-04-24T16:54:30.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:30 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3547945190' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:54:30.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:30 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/4173581614' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]: dispatch 2024-04-24T16:54:30.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:30 smithi156 ceph-mon[65097]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]: dispatch 2024-04-24T16:54:31.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:30 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3547945190' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:54:31.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:30 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/4173581614' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]: dispatch 2024-04-24T16:54:31.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:30 smithi070 ceph-mon[81602]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]: dispatch 2024-04-24T16:54:31.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:30 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3547945190' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T16:54:31.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:30 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/4173581614' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]: dispatch 2024-04-24T16:54:31.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:30 smithi070 ceph-mon[86496]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]: dispatch 2024-04-24T16:54:31.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:31 smithi156 ceph-mon[65097]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:54:31.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:31 smithi156 ceph-mon[65097]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]': finished 2024-04-24T16:54:31.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:31 smithi156 ceph-mon[65097]: osdmap e157: 8 total, 8 up, 8 in 2024-04-24T16:54:31.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:31 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2102277259' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4085606071"}]: dispatch 2024-04-24T16:54:32.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[86496]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:54:32.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[86496]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]': finished 2024-04-24T16:54:32.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[86496]: osdmap e157: 8 total, 8 up, 8 in 2024-04-24T16:54:32.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2102277259' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4085606071"}]: dispatch 2024-04-24T16:54:32.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[81602]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-24T16:54:32.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[81602]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/1265158380"}]': finished 2024-04-24T16:54:32.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[81602]: osdmap e157: 8 total, 8 up, 8 in 2024-04-24T16:54:32.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:31 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2102277259' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4085606071"}]: dispatch 2024-04-24T16:54:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2102277259' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4085606071"}]': finished 2024-04-24T16:54:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[86496]: osdmap e158: 8 total, 8 up, 8 in 2024-04-24T16:54:33.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/990581523' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]: dispatch 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[86496]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]: dispatch 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2102277259' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4085606071"}]': finished 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[81602]: osdmap e158: 8 total, 8 up, 8 in 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/990581523' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]: dispatch 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[81602]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]: dispatch 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:33.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:33.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:32 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2102277259' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/4085606071"}]': finished 2024-04-24T16:54:33.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:32 smithi156 ceph-mon[65097]: osdmap e158: 8 total, 8 up, 8 in 2024-04-24T16:54:33.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:32 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/990581523' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]: dispatch 2024-04-24T16:54:33.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:32 smithi156 ceph-mon[65097]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]: dispatch 2024-04-24T16:54:33.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:54:33.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[86496]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-04-24T16:54:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[86496]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]': finished 2024-04-24T16:54:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[86496]: osdmap e159: 8 total, 8 up, 8 in 2024-04-24T16:54:34.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1466010695' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1686646272"}]: dispatch 2024-04-24T16:54:34.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[81602]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-04-24T16:54:34.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[81602]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]': finished 2024-04-24T16:54:34.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[81602]: osdmap e159: 8 total, 8 up, 8 in 2024-04-24T16:54:34.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:33 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1466010695' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1686646272"}]: dispatch 2024-04-24T16:54:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:33 smithi156 ceph-mon[65097]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-04-24T16:54:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:33 smithi156 ceph-mon[65097]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6800/1686646272"}]': finished 2024-04-24T16:54:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:33 smithi156 ceph-mon[65097]: osdmap e159: 8 total, 8 up, 8 in 2024-04-24T16:54:34.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:33 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1466010695' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1686646272"}]: dispatch 2024-04-24T16:54:35.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:34 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1466010695' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1686646272"}]': finished 2024-04-24T16:54:35.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:34 smithi070 ceph-mon[86496]: osdmap e160: 8 total, 8 up, 8 in 2024-04-24T16:54:35.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:34 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/4129441331' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2084120879"}]: dispatch 2024-04-24T16:54:35.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:34 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1466010695' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1686646272"}]': finished 2024-04-24T16:54:35.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:34 smithi070 ceph-mon[81602]: osdmap e160: 8 total, 8 up, 8 in 2024-04-24T16:54:35.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:34 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/4129441331' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2084120879"}]: dispatch 2024-04-24T16:54:35.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:34 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1466010695' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:6801/1686646272"}]': finished 2024-04-24T16:54:35.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:34 smithi156 ceph-mon[65097]: osdmap e160: 8 total, 8 up, 8 in 2024-04-24T16:54:35.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:34 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/4129441331' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2084120879"}]: dispatch 2024-04-24T16:54:36.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[86496]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:54:36.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/4129441331' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2084120879"}]': finished 2024-04-24T16:54:36.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[86496]: osdmap e161: 8 total, 8 up, 8 in 2024-04-24T16:54:36.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/4184334622' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]: dispatch 2024-04-24T16:54:36.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[86496]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]: dispatch 2024-04-24T16:54:36.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[81602]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:54:36.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/4129441331' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2084120879"}]': finished 2024-04-24T16:54:36.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[81602]: osdmap e161: 8 total, 8 up, 8 in 2024-04-24T16:54:36.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/4184334622' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]: dispatch 2024-04-24T16:54:36.045 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:35 smithi070 ceph-mon[81602]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]: dispatch 2024-04-24T16:54:36.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:35 smithi156 ceph-mon[65097]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail 2024-04-24T16:54:36.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:35 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/4129441331' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/2084120879"}]': finished 2024-04-24T16:54:36.163 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:35 smithi156 ceph-mon[65097]: osdmap e161: 8 total, 8 up, 8 in 2024-04-24T16:54:36.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:35 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/4184334622' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]: dispatch 2024-04-24T16:54:36.164 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:35 smithi156 ceph-mon[65097]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]: dispatch 2024-04-24T16:54:36.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:36 smithi070 ceph-mon[81602]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]': finished 2024-04-24T16:54:36.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:36 smithi070 ceph-mon[81602]: osdmap e162: 8 total, 8 up, 8 in 2024-04-24T16:54:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:54:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:54:36] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-04-24T16:54:36.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:36 smithi070 ceph-mon[86496]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]': finished 2024-04-24T16:54:36.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:36 smithi070 ceph-mon[86496]: osdmap e162: 8 total, 8 up, 8 in 2024-04-24T16:54:36.808 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:54:37.141 INFO:teuthology.orchestra.run.smithi070.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed." 2024-04-24T16:54:37.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:36 smithi156 ceph-mon[65097]: from='client.? ' entity='client.iscsi.foo.smithi070.gcoqkr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.70:0/573744463"}]': finished 2024-04-24T16:54:37.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:36 smithi156 ceph-mon[65097]: osdmap e162: 8 total, 8 up, 8 in 2024-04-24T16:54:37.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:37 smithi070 ceph-mon[81602]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:37.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:37 smithi070 ceph-mon[81602]: from='client.64852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:37.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:37 smithi070 ceph-mon[81602]: from='client.64858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:37.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:37 smithi070 ceph-mon[86496]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:37.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:37 smithi070 ceph-mon[86496]: from='client.64852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:37.794 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:37 smithi070 ceph-mon[86496]: from='client.64858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:37.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:54:38.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:37 smithi156 ceph-mon[65097]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:38.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:37 smithi156 ceph-mon[65097]: from='client.64852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:38.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:37 smithi156 ceph-mon[65097]: from='client.64858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:40.007 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:39 smithi070 ceph-mon[86496]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-04-24T16:54:40.007 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:54:40.007 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (11m) 72s ago 33m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:54:40.007 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (10m) 72s ago 33m 80.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:54:40.007 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 72s ago 32m 66.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:54:40.007 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (12m) 72s ago 37m 455M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (22m) 72s ago 39m 515M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (9m) 72s ago 39m 58.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (10m) 72s ago 37m 53.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (8m) 72s ago 38m 50.8M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (12m) 72s ago 34m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (12m) 72s ago 33m 15.4M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (7m) 72s ago 37m 61.4M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (6m) 72s ago 37m 64.1M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (5m) 72s ago 36m 51.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (4m) 72s ago 36m 59.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (4m) 72s ago 35m 60.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (3m) 72s ago 35m 50.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (3m) 72s ago 35m 50.2M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:54:40.008 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (2m) 72s ago 34m 54.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff e2d46d53eba4 2024-04-24T16:54:40.009 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (11m) 72s ago 34m 75.1M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:54:40.009 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (81s) 72s ago 32m 74.8M - 19.0.0-3285-g5904d294 8bb2087275ff 8aa1086a1f79 2024-04-24T16:54:40.009 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (76s) 72s ago 32m 74.6M - 19.0.0-3285-g5904d294 8bb2087275ff f9d108c6acab 2024-04-24T16:54:40.009 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:39 smithi070 ceph-mon[81602]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-04-24T16:54:40.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:39 smithi156 ceph-mon[65097]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-04-24T16:54:40.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions' 2024-04-24T16:54:41.018 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:40 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:41.019 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:40 smithi070 ceph-mon[86496]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:41.019 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:40 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:41.019 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:40 smithi070 ceph-mon[81602]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:41.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:40 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:41.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:40 smithi156 ceph-mon[65097]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:54:42.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:41 smithi070 ceph-mon[81602]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-24T16:54:42.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:41 smithi070 ceph-mon[86496]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-24T16:54:42.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:41 smithi156 ceph-mon[65097]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-24T16:54:42.856 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:54:42.856 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:54:42.856 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:54:42.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:42.856 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 8 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 15 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:54:42.857 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:54:43.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-24T16:54:43.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:43 smithi070 ceph-mon[86496]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:43.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:43 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/3438498131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:43.793 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:43 smithi070 ceph-mon[81602]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:43.794 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:43 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/3438498131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:43 smithi156 ceph-mon[65097]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:44.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:43 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/3438498131' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:54:45.434 INFO:teuthology.orchestra.run.smithi070.stdout:wait for servicemap items w/ changing names to refresh 2024-04-24T16:54:45.980 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:45 smithi070 ceph-mon[86496]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:54:45.981 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:45 smithi070 ceph-mon[81602]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:54:46.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'sleep 60' 2024-04-24T16:54:46.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:45 smithi156 ceph-mon[65097]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-24T16:54:46.625 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:54:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:54:46] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-04-24T16:54:48.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:47 smithi070 ceph-mon[81602]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-04-24T16:54:48.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:48.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:47 smithi070 ceph-mon[86496]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-04-24T16:54:48.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:48.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:47 smithi156 ceph-mon[65097]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-04-24T16:54:48.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:54:50.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:49 smithi070 ceph-mon[81602]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:50.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:49 smithi070 ceph-mon[86496]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:50.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:49 smithi156 ceph-mon[65097]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:51.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:50 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:51.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:50 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:51.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:50 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:54:52.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:51 smithi070 ceph-mon[86496]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:52.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:51 smithi070 ceph-mon[81602]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:52.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:51 smithi156 ceph-mon[65097]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:54.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:53 smithi070 ceph-mon[86496]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:54.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:53 smithi070 ceph-mon[81602]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:54.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:53 smithi156 ceph-mon[65097]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:56.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:55 smithi070 ceph-mon[86496]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:56.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:55 smithi070 ceph-mon[81602]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:56.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:55 smithi156 ceph-mon[65097]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:54:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:54:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:54:56] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-04-24T16:54:58.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:57 smithi070 ceph-mon[81602]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:58.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:57 smithi070 ceph-mon[86496]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:54:58.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:57 smithi156 ceph-mon[65097]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:00.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:54:59 smithi070 ceph-mon[86496]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:00.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:54:59 smithi070 ceph-mon[81602]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:00.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:54:59 smithi156 ceph-mon[65097]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:01.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:00 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:01.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:00 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:01.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:00 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:02.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:01 smithi070 ceph-mon[86496]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:02.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:01 smithi070 ceph-mon[81602]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:02.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:01 smithi156 ceph-mon[65097]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:03.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:03.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:03.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:04.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:03 smithi070 ceph-mon[86496]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:04.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:03 smithi070 ceph-mon[81602]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:04.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:03 smithi156 ceph-mon[65097]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:06.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:05 smithi070 ceph-mon[86496]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:06.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:05 smithi070 ceph-mon[81602]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:06.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:05 smithi156 ceph-mon[65097]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:55:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:55:06] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-04-24T16:55:08.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:07 smithi070 ceph-mon[86496]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:08.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:07 smithi070 ceph-mon[81602]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:08.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:07 smithi156 ceph-mon[65097]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:10.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:09 smithi070 ceph-mon[81602]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:10.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:09 smithi070 ceph-mon[86496]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:10.161 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:09 smithi156 ceph-mon[65097]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:11.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:10 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:11.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:10 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:11.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:10 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:12.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:11 smithi070 ceph-mon[86496]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:12.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:11 smithi070 ceph-mon[81602]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:12.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:11 smithi156 ceph-mon[65097]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:14.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:13 smithi070 ceph-mon[81602]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:14.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:13 smithi070 ceph-mon[86496]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:14.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:13 smithi156 ceph-mon[65097]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:16.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:15 smithi070 ceph-mon[81602]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:16.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:15 smithi070 ceph-mon[86496]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:16.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:15 smithi156 ceph-mon[65097]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:16.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:55:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:55:16] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-04-24T16:55:18.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:17 smithi070 ceph-mon[86496]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:18.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:17 smithi070 ceph-mon[81602]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:18.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:17 smithi156 ceph-mon[65097]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:18.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:20.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[86496]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[81602]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:55:20.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:19 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:55:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:19 smithi156 ceph-mon[65097]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T16:55:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T16:55:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T16:55:20.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:19 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' 2024-04-24T16:55:21.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:20 smithi070 ceph-mon[86496]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-04-24T16:55:21.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:20 smithi070 ceph-mon[86496]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-04-24T16:55:21.043 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:20 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:21.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:20 smithi070 ceph-mon[81602]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-04-24T16:55:21.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:20 smithi070 ceph-mon[81602]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-04-24T16:55:21.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:20 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:21.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:20 smithi156 ceph-mon[65097]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-04-24T16:55:21.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:20 smithi156 ceph-mon[65097]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-04-24T16:55:21.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:20 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:23.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:23 smithi156 ceph-mon[65097]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:23.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:23 smithi070 ceph-mon[86496]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:23.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:23 smithi070 ceph-mon[81602]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:25.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:25 smithi156 ceph-mon[65097]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:25.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:25 smithi070 ceph-mon[81602]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:25.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:25 smithi070 ceph-mon[86496]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:26.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:55:26 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:55:26] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-04-24T16:55:27.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:27 smithi156 ceph-mon[65097]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:27.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:27 smithi070 ceph-mon[86496]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:27.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:27 smithi070 ceph-mon[81602]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:29.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:29 smithi156 ceph-mon[65097]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:29.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:29 smithi070 ceph-mon[86496]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:29.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:29 smithi070 ceph-mon[81602]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-24T16:55:31.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:31 smithi156 ceph-mon[65097]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:55:31.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:31 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:31.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:31 smithi070 ceph-mon[86496]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:55:31.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:31 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:31.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:31 smithi070 ceph-mon[81602]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-24T16:55:31.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:31 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:32.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:32 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:32.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:32 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:32.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:32 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:33.410 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:33 smithi156 ceph-mon[65097]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:33.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:33 smithi070 ceph-mon[81602]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:33.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:33 smithi070 ceph-mon[86496]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:35.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:35 smithi070 ceph-mon[86496]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:35.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:35 smithi070 ceph-mon[81602]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:35.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:35 smithi156 ceph-mon[65097]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:36.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:55:36 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:55:36] "GET /metrics HTTP/1.1" 200 38488 "" "Prometheus/2.43.0" 2024-04-24T16:55:37.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:37 smithi070 ceph-mon[86496]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:37.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:37 smithi070 ceph-mon[81602]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:37.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:37 smithi156 ceph-mon[65097]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:39.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:39 smithi070 ceph-mon[81602]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:39.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:39 smithi070 ceph-mon[86496]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:39.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:39 smithi156 ceph-mon[65097]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:41.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:41 smithi070 ceph-mon[81602]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:41.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:41 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:41.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:41 smithi070 ceph-mon[86496]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:41.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:41 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:41.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:41 smithi156 ceph-mon[65097]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:41.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:41 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:43.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:43 smithi070 ceph-mon[86496]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:43.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:43 smithi070 ceph-mon[81602]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:43.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:43 smithi156 ceph-mon[65097]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:45.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:45 smithi070 ceph-mon[81602]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:45.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:45 smithi070 ceph-mon[86496]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:45.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:45 smithi156 ceph-mon[65097]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:46.794 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:55:46 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:55:46] "GET /metrics HTTP/1.1" 200 38488 "" "Prometheus/2.43.0" 2024-04-24T16:55:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:47 smithi070 ceph-mon[86496]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:47.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:47 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:47.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:47 smithi070 ceph-mon[81602]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:47.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:47 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:47.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:47 smithi156 ceph-mon[65097]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:47.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:47 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:55:48.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ps' 2024-04-24T16:55:49.454 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:49 smithi070 ceph-mon[86496]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:49.455 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:49 smithi070 ceph-mon[81602]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:49.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:49 smithi156 ceph-mon[65097]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:55:50.462 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T16:55:50.462 INFO:teuthology.orchestra.run.smithi070.stdout:alertmanager.a smithi070 *:9093,9094 running (12m) 2m ago 34m 25.5M - 0.25.0 c8568f914cd2 a84220aabb99 2024-04-24T16:55:50.462 INFO:teuthology.orchestra.run.smithi070.stdout:grafana.a smithi156 *:3000 running (12m) 2m ago 34m 80.2M - 9.4.12 2bacad6d85d8 135413e06c85 2024-04-24T16:55:50.462 INFO:teuthology.orchestra.run.smithi070.stdout:iscsi.foo.smithi070.gcoqkr smithi070 error 2m ago 33m 66.1M - 3.5 e1d6a67b021e 38cbed88e822 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi156 *:8443,9283,8765 running (13m) 2m ago 38m 455M - 19.0.0-3285-g5904d294 8bb2087275ff 45888cc21913 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi070 *:8443,9283,8765 running (23m) 2m ago 40m 515M - 19.0.0-3285-g5904d294 8bb2087275ff e33f2bccae2e 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (10m) 2m ago 40m 58.9M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 538ac8b89d6a 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi156 running (11m) 2m ago 39m 53.5M 2048M 19.0.0-3285-g5904d294 8bb2087275ff 421c34aad07f 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi070 running (10m) 2m ago 39m 50.8M 2048M 19.0.0-3285-g5904d294 8bb2087275ff ea92875c6eec 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.a smithi070 *:9100 running (13m) 2m ago 35m 16.3M - 1.5.0 0da6a335fe13 88827079f0f6 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:node-exporter.b smithi156 *:9100 running (13m) 2m ago 35m 15.4M - 1.5.0 0da6a335fe13 9770cba17afb 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (8m) 2m ago 38m 61.4M 2992M 19.0.0-3285-g5904d294 8bb2087275ff fab1f0308200 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi070 running (7m) 2m ago 38m 64.1M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 3a6552201d2f 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi070 running (6m) 2m ago 37m 51.9M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 161808df2d43 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (5m) 2m ago 37m 59.7M 2992M 19.0.0-3285-g5904d294 8bb2087275ff 1c5293f7dc57 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi156 running (5m) 2m ago 37m 60.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 8b08bde60952 2024-04-24T16:55:50.463 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi156 running (4m) 2m ago 36m 50.4M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 4ab2caf74886 2024-04-24T16:55:50.464 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi156 running (4m) 2m ago 36m 50.2M 3248M 19.0.0-3285-g5904d294 8bb2087275ff 2972333b52bf 2024-04-24T16:55:50.464 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi156 running (3m) 2m ago 35m 54.0M 3248M 19.0.0-3285-g5904d294 8bb2087275ff e2d46d53eba4 2024-04-24T16:55:50.464 INFO:teuthology.orchestra.run.smithi070.stdout:prometheus.a smithi156 *:9095 running (12m) 2m ago 35m 75.1M - 2.43.0 a07b618ecd1d f240eb7d50fa 2024-04-24T16:55:50.464 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi070.vhdldq smithi070 *:8000 running (2m) 2m ago 33m 74.8M - 19.0.0-3285-g5904d294 8bb2087275ff 8aa1086a1f79 2024-04-24T16:55:50.464 INFO:teuthology.orchestra.run.smithi070.stdout:rgw.foo.smithi156.pmtzwp smithi156 *:8000 running (2m) 2m ago 33m 74.6M - 19.0.0-3285-g5904d294 8bb2087275ff f9d108c6acab 2024-04-24T16:55:51.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions' 2024-04-24T16:55:51.409 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:51 smithi070 ceph-mon[86496]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:51.410 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:51 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:51.410 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:51 smithi070 ceph-mon[81602]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:51.410 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:51 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:51.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:51 smithi156 ceph-mon[65097]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:51.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:51 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:55:52.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:52 smithi070 ceph-mon[86496]: from='client.64876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:55:52.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:52 smithi070 ceph-mon[81602]: from='client.64876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:55:52.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:52 smithi156 ceph-mon[65097]: from='client.64876 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:55:53.243 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": { 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 3 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": { 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": { 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 8 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw": { 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 2 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "overall": { 2024-04-24T16:55:53.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)": 15 2024-04-24T16:55:53.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-24T16:55:53.245 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:55:53.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:53 smithi070 ceph-mon[86496]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:53.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:53 smithi070 ceph-mon[81602]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:53.561 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:53 smithi156 ceph-mon[65097]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:53.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch upgrade status' 2024-04-24T16:55:54.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:54 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1030454259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:55:54.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:54 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1030454259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:55:54.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:54 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1030454259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:55:55.405 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:55 smithi070 ceph-mon[86496]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:55.406 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:55 smithi070 ceph-mon[81602]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:55.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:55 smithi156 ceph-mon[65097]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:55.853 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a", 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "in_progress": true, 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "services_complete": [ 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "osd", 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "rgw", 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "mon", 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr" 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-24T16:55:55.854 INFO:teuthology.orchestra.run.smithi070.stdout: "progress": "15/21 daemons upgraded", 2024-04-24T16:55:55.855 INFO:teuthology.orchestra.run.smithi070.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed.", 2024-04-24T16:55:55.855 INFO:teuthology.orchestra.run.smithi070.stdout: "is_paused": true 2024-04-24T16:55:55.855 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:55:56.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph health detail' 2024-04-24T16:55:56.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:55:56 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:55:56] "GET /metrics HTTP/1.1" 200 38483 "" "Prometheus/2.43.0" 2024-04-24T16:55:57.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:57 smithi070 ceph-mon[81602]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:57.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:57 smithi070 ceph-mon[81602]: from='client.64888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:55:57.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:57 smithi070 ceph-mon[86496]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:57.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:57 smithi070 ceph-mon[86496]: from='client.64888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:55:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:57 smithi156 ceph-mon[65097]: pgmap v410: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:55:57.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:57 smithi156 ceph-mon[65097]: from='client.64888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout:HEALTH_WARN 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state']; Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state'] 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout: daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in error state 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi070.gcoqkr on host smithi070 failed. 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrade daemon: iscsi.foo.smithi070.gcoqkr: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi070.gcoqkr ... 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout:Creating ceph-iscsi config... 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout:Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/iscsi-gateway.cfg 2024-04-24T16:55:58.641 INFO:teuthology.orchestra.run.smithi070.stdout:Write file: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/iscsi.foo.smithi070.gcoqkr/tcmu-runner-entrypoint.sh 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2dbe31be9a\x2d0255\x2d11ef\x2dbc93\x2dc7b262605968.slice/ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl start ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout:Traceback (most recent call last): 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: return _run_code(code, main_globals, None, 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: exec(code, run_globals) 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3117, in command_deploy_from 2024-04-24T16:55:58.642 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3152, in _common_deploy 2024-04-24T16:55:58.643 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3172, in _deploy_daemon_container 2024-04-24T16:55:58.643 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1087, in deploy_daemon 2024-04-24T16:55:58.643 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 1223, in deploy_daemon_units 2024-04-24T16:55:58.643 INFO:teuthology.orchestra.run.smithi070.stdout: File "/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T16:55:58.643 INFO:teuthology.orchestra.run.smithi070.stdout:RuntimeError: Failed command: systemctl start ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr: Job for ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service failed because the control process exited with error code. 2024-04-24T16:55:58.643 INFO:teuthology.orchestra.run.smithi070.stdout:See "systemctl status ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" and "journalctl -xeu ceph-be31be9a-0255-11ef-bc93-c7b262605968@iscsi.foo.smithi070.gcoqkr.service" for details. 2024-04-24T16:55:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:59 smithi070 ceph-mon[86496]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:59.293 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:55:59 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2430245283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:55:59.293 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:59 smithi070 ceph-mon[81602]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:59.294 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:55:59 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2430245283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:55:59.341 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-24T16:55:59.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:59 smithi156 ceph-mon[65097]: pgmap v411: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-24T16:55:59.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:55:59 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2430245283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-24T16:56:01.346 INFO:teuthology.orchestra.run.smithi070.stdout:true 2024-04-24T16:56:01.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:01 smithi070 ceph-mon[86496]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:01.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:01 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:56:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:01 smithi070 ceph-mon[81602]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:01.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:01 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:56:01.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:01 smithi156 ceph-mon[65097]: pgmap v412: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:01.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:01 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:56:01.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-24T16:56:02.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:02 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/1369985221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:56:02.476 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:02 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:56:02.476 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:02 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/1369985221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:56:02.476 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:02 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:56:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:02 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/1369985221' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:56:02.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:02 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:56:03.499 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:03 smithi070 ceph-mon[86496]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:03.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:03 smithi070 ceph-mon[81602]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:03.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:03 smithi156 ceph-mon[65097]: pgmap v413: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:03.912 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)" 2024-04-24T16:56:04.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:04 smithi070 ceph-mon[81602]: from='client.? 172.21.15.70:0/2846716201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:56:04.504 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:04 smithi070 ceph-mon[86496]: from='client.? 172.21.15.70:0/2846716201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:56:04.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -e sha1=5904d29475f5be602879d9fb26280e89b808d5cc -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-04-24T16:56:04.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:04 smithi156 ceph-mon[65097]: from='client.? 172.21.15.70:0/2846716201' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-24T16:56:05.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:05 smithi070 ceph-mon[81602]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:05.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:05 smithi070 ceph-mon[86496]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:05.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:05 smithi156 ceph-mon[65097]: pgmap v414: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:06.631 INFO:teuthology.orchestra.run.smithi070.stdout:osd 8 2m ago - 2024-04-24T16:56:06.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:06 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:56:06] "GET /metrics HTTP/1.1" 200 38484 "" "Prometheus/2.43.0" 2024-04-24T16:56:07.209 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T16:56:07.219 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-04-24T16:56:07.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-04-24T16:56:07.523 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:07 smithi070 ceph-mon[86496]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:07.523 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:07 smithi070 ceph-mon[81602]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:07.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:07 smithi156 ceph-mon[65097]: pgmap v415: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:08.501 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:08 smithi070 ceph-mon[86496]: from='client.64912 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:08.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:08 smithi070 ceph-mon[81602]: from='client.64912 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:08.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:08 smithi156 ceph-mon[65097]: from='client.64912 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:09.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:09 smithi070 ceph-mon[81602]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:09.544 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:09 smithi070 ceph-mon[86496]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:09.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:09 smithi156 ceph-mon[65097]: pgmap v416: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:09.921 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-24T16:56:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: "image": "quay.io/ceph/ceph", 2024-04-24T16:56:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: "registry": "quay.io", 2024-04-24T16:56:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: "bare_image": "ceph/ceph", 2024-04-24T16:56:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: "versions": [] 2024-04-24T16:56:09.921 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-24T16:56:10.491 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:10 smithi070 ceph-mon[86496]: from='client.54975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:10.492 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:10 smithi070 ceph-mon[86496]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:10.492 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:10 smithi070 ceph-mon[86496]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:56:10.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:10 smithi070 ceph-mon[81602]: from='client.54975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:10.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:10 smithi070 ceph-mon[81602]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:10.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:10 smithi070 ceph-mon[81602]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:56:10.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:10 smithi156 ceph-mon[65097]: from='client.54975 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:10.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:10 smithi156 ceph-mon[65097]: pgmap v417: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:10.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:10 smithi156 ceph-mon[65097]: from='client.64804 -' entity='client.iscsi.foo.smithi070.gcoqkr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-24T16:56:10.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-04-24T16:56:12.543 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:12 smithi070 ceph-mon[86496]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:12.544 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:12 smithi070 ceph-mon[81602]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:12.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:12 smithi156 ceph-mon[65097]: pgmap v418: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "16.2.0", 2024-04-24T16:56:13.748 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:13 smithi070 ceph-mon[86496]: from='client.54981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:13.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:13 smithi070 ceph-mon[81602]: from='client.54981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:13.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:13 smithi156 ceph-mon[65097]: from='client.54981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:14.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be31be9a-0255-11ef-bc93-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-04-24T16:56:14.618 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:14 smithi070 ceph-mon[86496]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:14.619 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:14 smithi070 ceph-mon[81602]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:14.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:14 smithi156 ceph-mon[65097]: pgmap v419: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-24T16:56:16.650 INFO:teuthology.orchestra.run.smithi070.stdout: "v16.2.2", 2024-04-24T16:56:16.650 INFO:teuthology.orchestra.run.smithi070.stdout: "v16.2.2-20210505", 2024-04-24T16:56:16.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:16 smithi070 ceph-mon[86496]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:16.788 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:16 smithi070 ceph-mon[86496]: from='client.54987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:16.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:16 smithi070 ceph-mon[81602]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:16.788 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:16 smithi070 ceph-mon[81602]: from='client.54987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:16.788 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:16 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: ::ffff:172.21.15.156 - - [24/Apr/2024:16:56:16] "GET /metrics HTTP/1.1" 200 38484 "" "Prometheus/2.43.0" 2024-04-24T16:56:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:16 smithi156 ceph-mon[65097]: pgmap v420: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-24T16:56:16.910 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:16 smithi156 ceph-mon[65097]: from='client.54987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T16:56:17.320 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-24T16:56:17.331 INFO:tasks.cephadm:Teardown begin 2024-04-24T16:56:17.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T16:56:17.357 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T16:56:17.390 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-24T16:56:17.390 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T16:56:17.413 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T16:56:17.448 INFO:tasks.cephadm:Stopping all daemons... 2024-04-24T16:56:17.448 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-24T16:56:17.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a 2024-04-24T16:56:17.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:17 smithi156 ceph-mon[65097]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:56:17.725 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-mon[86496]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:56:17.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-mon[81602]: from='mgr.54108 172.21.15.70:0/263763002' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-24T16:56:17.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:17 smithi070 systemd[1]: Stopping Ceph mon.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a[81598]: 2024-04-24T16:56:17.733+0000 7f9270ecc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:18.043 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a[81598]: 2024-04-24T16:56:17.733+0000 7f9270ecc700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:17 smithi070 podman[128190]: 2024-04-24 16:56:17.897127099 +0000 UTC m=+0.257627408 container died 538ac8b89d6a37ce41dc5e1c58a8e1025fc86c81012411a7c202e4f50c4b8dfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:56:17] ENGINE Bus STOPPING 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:56:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:56:17] ENGINE Bus STOPPED 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:56:17] ENGINE Bus STARTING 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: CherryPy Checker: 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: The Application mounted at '' has an empty config. 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:56:17] ENGINE Serving on http://:::9283 2024-04-24T16:56:18.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:17 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: [24/Apr/2024:16:56:17] ENGINE Bus STARTED 2024-04-24T16:56:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:18 smithi070 podman[128190]: 2024-04-24 16:56:18.250120493 +0000 UTC m=+0.610620803 container cleanup 538ac8b89d6a37ce41dc5e1c58a8e1025fc86c81012411a7c202e4f50c4b8dfb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-24T16:56:18.543 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 24 16:56:18 smithi070 bash[128190]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-a 2024-04-24T16:56:18.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.a.service' 2024-04-24T16:56:18.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:18.762 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-24T16:56:18.762 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-24T16:56:18.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.c 2024-04-24T16:56:19.113 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:18 smithi070 systemd[1]: Stopping Ceph mon.c for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:19.471 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c[86492]: 2024-04-24T16:56:19.112+0000 7fac57339700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:19.472 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:19 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c[86492]: 2024-04-24T16:56:19.112+0000 7fac57339700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-04-24T16:56:19.472 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:19 smithi070 podman[128323]: 2024-04-24 16:56:19.205074464 +0000 UTC m=+0.224366017 container died ea92875c6eecf9b15740d388637bb66b82e9dcc2aec03b0f53f46e29a8581ead (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T16:56:19.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:19 smithi070 podman[128323]: 2024-04-24 16:56:19.562771607 +0000 UTC m=+0.582063152 container cleanup ea92875c6eecf9b15740d388637bb66b82e9dcc2aec03b0f53f46e29a8581ead (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-24T16:56:19.793 INFO:journalctl@ceph.mon.c.smithi070.stdout:Apr 24 16:56:19 smithi070 bash[128323]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-c 2024-04-24T16:56:19.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.c.service' 2024-04-24T16:56:20.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:20.024 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-24T16:56:20.024 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-24T16:56:20.024 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.b 2024-04-24T16:56:20.376 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:20 smithi156 systemd[1]: Stopping Ceph mon.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:20.660 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:20 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b[65093]: 2024-04-24T16:56:20.374+0000 7f6f7d442700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:20.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:20 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b[65093]: 2024-04-24T16:56:20.374+0000 7f6f7d442700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-04-24T16:56:20.661 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:20 smithi156 podman[99303]: 2024-04-24 16:56:20.569425933 +0000 UTC m=+0.345283379 container died 421c34aad07fdc440ee676357a13929e4ccf510fec6c416dd08357eec992853e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:56:21.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:21 smithi156 podman[99303]: 2024-04-24 16:56:21.066839784 +0000 UTC m=+0.842697229 container cleanup 421c34aad07fdc440ee676357a13929e4ccf510fec6c416dd08357eec992853e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-24T16:56:21.160 INFO:journalctl@ceph.mon.b.smithi156.stdout:Apr 24 16:56:21 smithi156 bash[99303]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mon-b 2024-04-24T16:56:21.407 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mon.b.service' 2024-04-24T16:56:21.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:21.450 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-24T16:56:21.450 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-24T16:56:21.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y 2024-04-24T16:56:21.767 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:21 smithi070 systemd[1]: Stopping Ceph mgr.y for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:22.043 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:21 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:56:21.766+0000 7f22ba5da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:22.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:21 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y[60572]: 2024-04-24T16:56:21.766+0000 7f22ba5da700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-24T16:56:22.044 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:21 smithi070 podman[128444]: 2024-04-24 16:56:21.798239822 +0000 UTC m=+0.153624070 container died e33f2bccae2e2df01228af6e8d2a17bf96bb5c609e7f882f34d09716568bc6a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5) 2024-04-24T16:56:22.436 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:22 smithi070 podman[128444]: 2024-04-24 16:56:22.209726175 +0000 UTC m=+0.565110412 container cleanup e33f2bccae2e2df01228af6e8d2a17bf96bb5c609e7f882f34d09716568bc6a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T16:56:22.436 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 24 16:56:22 smithi070 bash[128444]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-y 2024-04-24T16:56:22.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-24T16:56:22.612 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:22.612 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-24T16:56:22.612 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-24T16:56:22.612 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x 2024-04-24T16:56:22.910 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:56:22 smithi156 systemd[1]: Stopping Ceph mgr.x for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:23.197 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:56:22 smithi156 podman[99426]: 2024-04-24 16:56:22.944344681 +0000 UTC m=+0.135844108 container died 45888cc21913fa855f0bbe574df10e47910e16ac210b710ecc196391f79e3a61 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T16:56:23.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:56:23 smithi156 podman[99426]: 2024-04-24 16:56:23.271318181 +0000 UTC m=+0.462817612 container cleanup 45888cc21913fa855f0bbe574df10e47910e16ac210b710ecc196391f79e3a61 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-24T16:56:23.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:56:23 smithi156 bash[99426]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x 2024-04-24T16:56:23.660 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:56:23 smithi156 podman[99439]: 2024-04-24 16:56:23.439378731 +0000 UTC m=+0.496808132 container remove 45888cc21913fa855f0bbe574df10e47910e16ac210b710ecc196391f79e3a61 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T16:56:23.661 INFO:journalctl@ceph.mgr.x.smithi156.stdout:Apr 24 16:56:23 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:56:23.689 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-24T16:56:23.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:23.723 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-24T16:56:23.723 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-24T16:56:23.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.0 2024-04-24T16:56:24.030 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:23 smithi070 systemd[1]: Stopping Ceph osd.0 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:24.030 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:23.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:24.031 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:23.552Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:24.293 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[93809]: 2024-04-24T16:56:24.029+0000 7f5ab2b82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:24.293 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[93809]: 2024-04-24T16:56:24.029+0000 7f5ab2b82700 -1 osd.0 162 *** Got signal Terminated *** 2024-04-24T16:56:24.293 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0[93809]: 2024-04-24T16:56:24.029+0000 7f5ab2b82700 -1 osd.0 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:56:29.838 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:29 smithi070 podman[128566]: 2024-04-24 16:56:29.571953239 +0000 UTC m=+5.655821066 container died fab1f03082000ea2ad8b2caad26984eca37e05843a151cc8e0eb73e0bbce847b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:56:30.222 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:29 smithi070 podman[128566]: 2024-04-24 16:56:29.929994746 +0000 UTC m=+6.013862607 container cleanup fab1f03082000ea2ad8b2caad26984eca37e05843a151cc8e0eb73e0bbce847b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T16:56:30.223 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:29 smithi070 bash[128566]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0 2024-04-24T16:56:30.543 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:30 smithi070 podman[128583]: 2024-04-24 16:56:30.223123761 +0000 UTC m=+0.648122645 container remove fab1f03082000ea2ad8b2caad26984eca37e05843a151cc8e0eb73e0bbce847b (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-24T16:56:30.818 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:30 smithi070 podman[128675]: 2024-04-24 16:56:30.549145656 +0000 UTC m=+0.113538101 container create a3d8072f68dc040871d2f30004bf2382a31289dda01567fba653fa1b3fd3d61e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-24T16:56:30.818 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:30 smithi070 podman[128675]: 2024-04-24 16:56:30.453686904 +0000 UTC m=+0.018079321 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:56:30.818 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:30 smithi070 podman[128675]: 2024-04-24 16:56:30.760090062 +0000 UTC m=+0.324482466 container init a3d8072f68dc040871d2f30004bf2382a31289dda01567fba653fa1b3fd3d61e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-24T16:56:30.818 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:30 smithi070 podman[128675]: 2024-04-24 16:56:30.763317669 +0000 UTC m=+0.327710081 container start a3d8072f68dc040871d2f30004bf2382a31289dda01567fba653fa1b3fd3d61e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-24T16:56:30.818 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:30 smithi070 podman[128675]: 2024-04-24 16:56:30.818193798 +0000 UTC m=+0.382586205 container attach a3d8072f68dc040871d2f30004bf2382a31289dda01567fba653fa1b3fd3d61e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-24T16:56:31.087 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 24 16:56:31 smithi070 podman[128675]: 2024-04-24 16:56:31.056230947 +0000 UTC m=+0.620623351 container died a3d8072f68dc040871d2f30004bf2382a31289dda01567fba653fa1b3fd3d61e (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:56:31.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.0.service' 2024-04-24T16:56:31.685 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:31.685 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-24T16:56:31.686 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-24T16:56:31.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.1 2024-04-24T16:56:32.002 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:31 smithi070 systemd[1]: Stopping Ceph osd.1 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:32.258 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[98993]: 2024-04-24T16:56:32.001+0000 7faa28048700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:32.258 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[98993]: 2024-04-24T16:56:32.001+0000 7faa28048700 -1 osd.1 162 *** Got signal Terminated *** 2024-04-24T16:56:32.258 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:32 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1[98993]: 2024-04-24T16:56:32.001+0000 7faa28048700 -1 osd.1 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:56:34.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:33.552Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:34.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:33.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:34.044 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:33 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:33.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:34.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:34 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:34.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:34.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:34 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:34.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:37.794 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:37 smithi070 podman[128790]: 2024-04-24 16:56:37.540974506 +0000 UTC m=+5.635280131 container died 3a6552201d2f7ec683d5088b15253229c0026e9bbd27f213a43603c5b57737b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-24T16:56:38.177 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:37 smithi070 podman[128790]: 2024-04-24 16:56:37.943475822 +0000 UTC m=+6.037781429 container cleanup 3a6552201d2f7ec683d5088b15253229c0026e9bbd27f213a43603c5b57737b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-24T16:56:38.178 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:37 smithi070 bash[128790]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1 2024-04-24T16:56:38.488 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:38 smithi070 podman[128813]: 2024-04-24 16:56:38.178209503 +0000 UTC m=+0.632692183 container remove 3a6552201d2f7ec683d5088b15253229c0026e9bbd27f213a43603c5b57737b2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:56:38.749 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:38 smithi070 podman[128907]: 2024-04-24 16:56:38.39124741 +0000 UTC m=+0.017814612 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:56:38.749 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:38 smithi070 podman[128907]: 2024-04-24 16:56:38.503838669 +0000 UTC m=+0.130405852 container create 01e2facfe3b19726a568e8f3e21e5989a9de8faa153a1566a2af8cb4ef688b25 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-04-24T16:56:38.749 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:38 smithi070 podman[128907]: 2024-04-24 16:56:38.749162721 +0000 UTC m=+0.375729906 container init 01e2facfe3b19726a568e8f3e21e5989a9de8faa153a1566a2af8cb4ef688b25 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-24T16:56:39.043 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:38 smithi070 podman[128907]: 2024-04-24 16:56:38.752457805 +0000 UTC m=+0.379024992 container start 01e2facfe3b19726a568e8f3e21e5989a9de8faa153a1566a2af8cb4ef688b25 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:56:39.043 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:38 smithi070 podman[128907]: 2024-04-24 16:56:38.839987578 +0000 UTC m=+0.466554775 container attach 01e2facfe3b19726a568e8f3e21e5989a9de8faa153a1566a2af8cb4ef688b25 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:56:39.326 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 24 16:56:39 smithi070 podman[128907]: 2024-04-24 16:56:39.046422862 +0000 UTC m=+0.672990041 container died 01e2facfe3b19726a568e8f3e21e5989a9de8faa153a1566a2af8cb4ef688b25 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:56:39.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.1.service' 2024-04-24T16:56:39.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:39.647 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-24T16:56:39.647 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-24T16:56:39.648 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.2 2024-04-24T16:56:39.971 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:39 smithi070 systemd[1]: Stopping Ceph osd.2 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:40.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[105476]: 2024-04-24T16:56:39.970+0000 7f3f45c27700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:40.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[105476]: 2024-04-24T16:56:39.970+0000 7f3f45c27700 -1 osd.2 162 *** Got signal Terminated *** 2024-04-24T16:56:40.293 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:39 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2[105476]: 2024-04-24T16:56:39.970+0000 7f3f45c27700 -1 osd.2 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:56:44.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:43.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:44.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:43.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:44.044 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:43 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:43.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:44.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:44.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:44.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:44.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:44.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:44 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:44.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:45.629 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:45 smithi070 podman[129014]: 2024-04-24 16:56:45.342988278 +0000 UTC m=+5.480874717 container died 161808df2d438da4491f50c061f2211ed4d93331b1b32e31bfc9f400ce216375 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:56:45.629 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:45 smithi070 podman[129014]: 2024-04-24 16:56:45.629064248 +0000 UTC m=+5.766950687 container cleanup 161808df2d438da4491f50c061f2211ed4d93331b1b32e31bfc9f400ce216375 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T16:56:45.889 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:45 smithi070 bash[129014]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2 2024-04-24T16:56:46.252 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:45 smithi070 podman[129032]: 2024-04-24 16:56:45.889211175 +0000 UTC m=+0.540751448 container remove 161808df2d438da4491f50c061f2211ed4d93331b1b32e31bfc9f400ce216375 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-24T16:56:46.533 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:46 smithi070 podman[129134]: 2024-04-24 16:56:46.15612499 +0000 UTC m=+0.018625311 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:56:46.534 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:46 smithi070 podman[129134]: 2024-04-24 16:56:46.264956378 +0000 UTC m=+0.127456703 container create efb2dfd9382bd55116ce1021bb597b45a3d48e2cd2cb65f1941ee66efd2828bb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-24T16:56:46.534 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:46 smithi070 podman[129134]: 2024-04-24 16:56:46.458972274 +0000 UTC m=+0.321472596 container init efb2dfd9382bd55116ce1021bb597b45a3d48e2cd2cb65f1941ee66efd2828bb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:56:46.534 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:46 smithi070 podman[129134]: 2024-04-24 16:56:46.462187156 +0000 UTC m=+0.324687480 container start efb2dfd9382bd55116ce1021bb597b45a3d48e2cd2cb65f1941ee66efd2828bb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-24T16:56:46.534 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:46 smithi070 podman[129134]: 2024-04-24 16:56:46.533952866 +0000 UTC m=+0.396453189 container attach efb2dfd9382bd55116ce1021bb597b45a3d48e2cd2cb65f1941ee66efd2828bb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:56:46.793 INFO:journalctl@ceph.osd.2.smithi070.stdout:Apr 24 16:56:46 smithi070 podman[129134]: 2024-04-24 16:56:46.759508851 +0000 UTC m=+0.622009171 container died efb2dfd9382bd55116ce1021bb597b45a3d48e2cd2cb65f1941ee66efd2828bb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-24T16:56:47.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.2.service' 2024-04-24T16:56:47.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:47.368 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-24T16:56:47.368 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-24T16:56:47.368 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.3 2024-04-24T16:56:47.715 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:47 smithi070 systemd[1]: Stopping Ceph osd.3 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:48.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:56:47.714+0000 7f06b1197700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:48.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:56:47.714+0000 7f06b1197700 -1 osd.3 162 *** Got signal Terminated *** 2024-04-24T16:56:48.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:47 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:56:47.714+0000 7f06b1197700 -1 osd.3 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:56:50.411 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:50 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:50.061+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:51.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:50 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:56:50.652+0000 7f06a9783700 -1 osd.3 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.675376+0000 front 2024-04-24T16:56:27.675472+0000 (oldest deadline 2024-04-24T16:56:49.975475+0000) 2024-04-24T16:56:51.410 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:51.019+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:51.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:51 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:51.247+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:52.043 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:51 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:56:51.696+0000 7f06a9783700 -1 osd.3 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.675376+0000 front 2024-04-24T16:56:27.675472+0000 (oldest deadline 2024-04-24T16:56:49.975475+0000) 2024-04-24T16:56:52.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:52.248+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:52.411 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:52 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:52.047+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:53.027 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:52 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3[111922]: 2024-04-24T16:56:52.661+0000 7f06a9783700 -1 osd.3 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.675376+0000 front 2024-04-24T16:56:27.675472+0000 (oldest deadline 2024-04-24T16:56:49.975475+0000) 2024-04-24T16:56:53.293 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:53 smithi070 podman[129238]: 2024-04-24 16:56:53.027418147 +0000 UTC m=+5.436338420 container died 1c5293f7dc5747fbd7611883c01fac3dfc1cd66cd76e71c4963d05bee8a5b5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:56:53.410 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:53.081+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:53.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:53.272+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:53.649 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:53 smithi070 podman[129238]: 2024-04-24 16:56:53.406347649 +0000 UTC m=+5.815267925 container cleanup 1c5293f7dc5747fbd7611883c01fac3dfc1cd66cd76e71c4963d05bee8a5b5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T16:56:53.649 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:53 smithi070 bash[129238]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3 2024-04-24T16:56:53.649 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:53.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:53.649 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:53.649 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:53 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:53.910 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:56:53 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:56:53.702+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:56:53.966 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:53 smithi070 podman[129255]: 2024-04-24 16:56:53.64917521 +0000 UTC m=+0.614941791 container remove 1c5293f7dc5747fbd7611883c01fac3dfc1cd66cd76e71c4963d05bee8a5b5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T16:56:54.252 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:53 smithi070 podman[129349]: 2024-04-24 16:56:53.966530244 +0000 UTC m=+0.110103111 container create dc8fbf546f9502b49ca03422efbc74d745512dcc3bb9577d3fec059ff020137f (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T16:56:54.252 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:53 smithi070 podman[129349]: 2024-04-24 16:56:53.874540351 +0000 UTC m=+0.018113252 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:56:54.252 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:54 smithi070 podman[129349]: 2024-04-24 16:56:54.169042777 +0000 UTC m=+0.312615648 container init dc8fbf546f9502b49ca03422efbc74d745512dcc3bb9577d3fec059ff020137f (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-24T16:56:54.252 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:54 smithi070 podman[129349]: 2024-04-24 16:56:54.172489326 +0000 UTC m=+0.316062196 container start dc8fbf546f9502b49ca03422efbc74d745512dcc3bb9577d3fec059ff020137f (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T16:56:54.252 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:54 smithi070 podman[129349]: 2024-04-24 16:56:54.252110968 +0000 UTC m=+0.395683860 container attach dc8fbf546f9502b49ca03422efbc74d745512dcc3bb9577d3fec059ff020137f (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424) 2024-04-24T16:56:54.353 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:54.052+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:54.353 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:54.281+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:54.544 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:54 smithi070 podman[129349]: 2024-04-24 16:56:54.467444647 +0000 UTC m=+0.611017526 container died dc8fbf546f9502b49ca03422efbc74d745512dcc3bb9577d3fec059ff020137f (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T16:56:54.544 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:54.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:54.544 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:54.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:56:54.544 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:56:54 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:56:54.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:56:54.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:54.352+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:28.835168+0000 front 2024-04-24T16:56:28.835151+0000 (oldest deadline 2024-04-24T16:56:53.534901+0000) 2024-04-24T16:56:55.064 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 24 16:56:55 smithi070 podman[129377]: 2024-04-24 16:56:55.047270213 +0000 UTC m=+0.574307952 container remove dc8fbf546f9502b49ca03422efbc74d745512dcc3bb9577d3fec059ff020137f (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-24T16:56:55.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.3.service' 2024-04-24T16:56:55.068 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:56:54 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:56:54.685+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:56:55.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:56:55.103 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-24T16:56:55.103 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-24T16:56:55.104 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.4 2024-04-24T16:56:55.351 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:55.067+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:55.351 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:55.277+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:55.351 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:55 smithi156 systemd[1]: Stopping Ceph osd.4 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:56:55.351 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:55.350+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:28.835168+0000 front 2024-04-24T16:56:28.835151+0000 (oldest deadline 2024-04-24T16:56:53.534901+0000) 2024-04-24T16:56:55.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:55.439+0000 7f85ce80f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:56:55.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:55.439+0000 7f85ce80f700 -1 osd.4 162 *** Got signal Terminated *** 2024-04-24T16:56:55.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:55.439+0000 7f85ce80f700 -1 osd.4 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:56:56.024 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:56:55 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:56:55.665+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:56:56.358 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:56.023+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:56.358 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:56.260+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:56.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:56.357+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:28.835168+0000 front 2024-04-24T16:56:28.835151+0000 (oldest deadline 2024-04-24T16:56:53.534901+0000) 2024-04-24T16:56:56.992 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:56:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:56:56.698+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:56:57.273 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:56 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:56.992+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:57.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:57.362+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:28.835168+0000 front 2024-04-24T16:56:28.835151+0000 (oldest deadline 2024-04-24T16:56:53.534901+0000) 2024-04-24T16:56:57.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:57.273+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:57.965 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:56:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:56:57.696+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:56:58.253 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:57 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:57.964+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:58.661 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:58.314+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:28.835168+0000 front 2024-04-24T16:56:28.835151+0000 (oldest deadline 2024-04-24T16:56:53.534901+0000) 2024-04-24T16:56:58.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:58.251+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:58.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:58.251+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:56:59.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:56:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:56:58.700+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:56:59.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:58.933+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:56:59.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:58.933+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:56:59.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:59.284+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:28.835168+0000 front 2024-04-24T16:56:28.835151+0000 (oldest deadline 2024-04-24T16:56:53.534901+0000) 2024-04-24T16:56:59.661 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:56:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:56:59.284+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:33.535535+0000 front 2024-04-24T16:56:33.535486+0000 (oldest deadline 2024-04-24T16:56:58.835463+0000) 2024-04-24T16:56:59.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:59.221+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:56:59.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:56:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:56:59.221+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:00.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:56:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:56:59.710+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:00.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:59.929+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:00.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:56:59 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:56:59.929+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:00.660 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:57:00.298+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:28.835168+0000 front 2024-04-24T16:56:28.835151+0000 (oldest deadline 2024-04-24T16:56:53.534901+0000) 2024-04-24T16:57:00.661 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4[75900]: 2024-04-24T16:57:00.298+0000 7f85c6dfb700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:33.535535+0000 front 2024-04-24T16:56:33.535486+0000 (oldest deadline 2024-04-24T16:56:58.835463+0000) 2024-04-24T16:57:00.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:00.212+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:00.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:00.212+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:01.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:00.752+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:01.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:00 smithi156 podman[99569]: 2024-04-24 16:57:00.933785212 +0000 UTC m=+5.637182218 container died 8b08bde60952e0c95de10c9a86ed8ca4b291f0011b8bbc8226c56a53501d6b0a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-24T16:57:01.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:00.929+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:01.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:00 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:00.929+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:01.549 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:01 smithi156 podman[99569]: 2024-04-24 16:57:01.365233258 +0000 UTC m=+6.068630309 container cleanup 8b08bde60952e0c95de10c9a86ed8ca4b291f0011b8bbc8226c56a53501d6b0a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:57:01.549 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:01 smithi156 bash[99569]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4 2024-04-24T16:57:01.550 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:01.170+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:01.550 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:01.170+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:01.886 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:01 smithi156 podman[99586]: 2024-04-24 16:57:01.549878441 +0000 UTC m=+0.616079930 container remove 8b08bde60952e0c95de10c9a86ed8ca4b291f0011b8bbc8226c56a53501d6b0a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-24T16:57:01.886 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:01.719+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:01.886 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:01.719+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.401239+0000 front 2024-04-24T16:56:36.401218+0000 (oldest deadline 2024-04-24T16:57:01.101235+0000) 2024-04-24T16:57:01.887 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:01.618Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-04-24T16:57:01.887 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:01.621Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-04-24T16:57:01.887 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:01.622Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-04-24T16:57:01.887 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:01.623Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-04-24T16:57:01.887 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:01.624Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.70:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.70:8765: connect: connection refused" 2024-04-24T16:57:02.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:01.885+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:02.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:01 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:01.885+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:02.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:01 smithi156 podman[99678]: 2024-04-24 16:57:01.870324267 +0000 UTC m=+0.018275854 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:57:02.161 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:01 smithi156 podman[99678]: 2024-04-24 16:57:01.992593585 +0000 UTC m=+0.140545176 container create 1ab6e96fdc8e8c42d9f864082dccf0962768d2b8e6b79569b42c562a3c2e7e2d (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-24T16:57:02.567 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:02 smithi156 podman[99678]: 2024-04-24 16:57:02.270122456 +0000 UTC m=+0.418074044 container init 1ab6e96fdc8e8c42d9f864082dccf0962768d2b8e6b79569b42c562a3c2e7e2d (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-24T16:57:02.567 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:02 smithi156 podman[99678]: 2024-04-24 16:57:02.273303798 +0000 UTC m=+0.421255395 container start 1ab6e96fdc8e8c42d9f864082dccf0962768d2b8e6b79569b42c562a3c2e7e2d (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29) 2024-04-24T16:57:02.567 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:02 smithi156 podman[99678]: 2024-04-24 16:57:02.353361317 +0000 UTC m=+0.501312928 container attach 1ab6e96fdc8e8c42d9f864082dccf0962768d2b8e6b79569b42c562a3c2e7e2d (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True) 2024-04-24T16:57:02.567 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:02.165+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:02.567 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:02.165+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:02.873 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:02.751+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:02.873 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:02.751+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.401239+0000 front 2024-04-24T16:56:36.401218+0000 (oldest deadline 2024-04-24T16:57:01.101235+0000) 2024-04-24T16:57:02.873 INFO:journalctl@ceph.osd.4.smithi156.stdout:Apr 24 16:57:02 smithi156 podman[99678]: 2024-04-24 16:57:02.567072095 +0000 UTC m=+0.715023692 container died 1ab6e96fdc8e8c42d9f864082dccf0962768d2b8e6b79569b42c562a3c2e7e2d (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-24T16:57:03.123 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:02.886+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:03.123 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:02 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:02.886+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:03.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.4.service' 2024-04-24T16:57:03.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:57:03.190 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-24T16:57:03.190 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-24T16:57:03.190 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.5 2024-04-24T16:57:03.410 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:03 smithi156 systemd[1]: Stopping Ceph osd.5 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:03.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:03.181+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:03.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:03.181+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:03.860 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:03.535+0000 7f1c1d6d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:57:03.860 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:03.535+0000 7f1c1d6d3700 -1 osd.5 162 *** Got signal Terminated *** 2024-04-24T16:57:03.860 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:03.535+0000 7f1c1d6d3700 -1 osd.5 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:57:03.860 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:03.725+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:03.860 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:03.725+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.401239+0000 front 2024-04-24T16:56:36.401218+0000 (oldest deadline 2024-04-24T16:57:01.101235+0000) 2024-04-24T16:57:04.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:03.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:04.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:03.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:04.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:03 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:03.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:04.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:03.859+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:04.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:03 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:03.859+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:04.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:04.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:04.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:04.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:04.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:04 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:04.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:04.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:04.162+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:04.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:04.162+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:05.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:04.709+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:05.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:04.709+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.401239+0000 front 2024-04-24T16:56:36.401218+0000 (oldest deadline 2024-04-24T16:57:01.101235+0000) 2024-04-24T16:57:05.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:04.835+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:05.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:04 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:04.835+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:05.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:05.209+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:05.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:05.209+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:06.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:05.797+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:06.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:05.797+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:06.161 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:05.712+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:06.161 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:05 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:05.712+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.401239+0000 front 2024-04-24T16:56:36.401218+0000 (oldest deadline 2024-04-24T16:57:01.101235+0000) 2024-04-24T16:57:06.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:06.249+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:06.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:06.249+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:07.160 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:06.670+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:07.161 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:06.670+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.401239+0000 front 2024-04-24T16:56:36.401218+0000 (oldest deadline 2024-04-24T16:57:01.101235+0000) 2024-04-24T16:57:07.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:06.806+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:07.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:06.806+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:07.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:06 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:06.806+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:07.641 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:07.211+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:07.642 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:07.211+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:07.910 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:07.641+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:27.699903+0000 front 2024-04-24T16:56:27.699981+0000 (oldest deadline 2024-04-24T16:56:52.999844+0000) 2024-04-24T16:57:07.911 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:07.641+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.401239+0000 front 2024-04-24T16:56:36.401218+0000 (oldest deadline 2024-04-24T16:57:01.101235+0000) 2024-04-24T16:57:07.911 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5[80750]: 2024-04-24T16:57:07.641+0000 7f1c19cc7700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.402270+0000 front 2024-04-24T16:56:43.402379+0000 (oldest deadline 2024-04-24T16:57:07.502374+0000) 2024-04-24T16:57:07.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:07.767+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:07.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:07.767+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:07.911 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:07 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:07.767+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:08.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:08.243+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:08.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:08.243+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:09.061 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:08.758+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:09.061 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:08.758+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:09.061 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:08 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:08.758+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:09.353 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:09 smithi156 podman[99782]: 2024-04-24 16:57:09.060694599 +0000 UTC m=+5.644483437 container died 4ab2caf74886615c99d860b08045cf951aebeecd62c178d70602bda0f170b787 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-24T16:57:09.354 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:09.227+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:09.354 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:09.227+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:09.660 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:09 smithi156 podman[99782]: 2024-04-24 16:57:09.535506688 +0000 UTC m=+6.119295560 container cleanup 4ab2caf74886615c99d860b08045cf951aebeecd62c178d70602bda0f170b787 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-24T16:57:09.660 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:09 smithi156 bash[99782]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5 2024-04-24T16:57:10.156 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:09 smithi156 podman[99800]: 2024-04-24 16:57:09.795184985 +0000 UTC m=+0.721232360 container remove 4ab2caf74886615c99d860b08045cf951aebeecd62c178d70602bda0f170b787 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:57:10.156 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:09.740+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:10.156 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:09.740+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:10.156 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:09 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:09.740+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:10.411 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:10 smithi156 podman[99893]: 2024-04-24 16:57:10.059482046 +0000 UTC m=+0.017859732 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:57:10.411 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:10 smithi156 podman[99893]: 2024-04-24 16:57:10.162642106 +0000 UTC m=+0.121019773 container create ef8a613c04c570e4ddb0fb834de339619818150858291c0680a458a7ca216f95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-24T16:57:10.411 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:10 smithi156 podman[99893]: 2024-04-24 16:57:10.399660914 +0000 UTC m=+0.358038583 container init ef8a613c04c570e4ddb0fb834de339619818150858291c0680a458a7ca216f95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T16:57:10.411 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:10 smithi156 podman[99893]: 2024-04-24 16:57:10.402883883 +0000 UTC m=+0.361261554 container start ef8a613c04c570e4ddb0fb834de339619818150858291c0680a458a7ca216f95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-24T16:57:10.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:10.232+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:10.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:10.232+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:10.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:10.232+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:10.757 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:10 smithi156 podman[99893]: 2024-04-24 16:57:10.49887342 +0000 UTC m=+0.457251107 container attach ef8a613c04c570e4ddb0fb834de339619818150858291c0680a458a7ca216f95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-24T16:57:10.757 INFO:journalctl@ceph.osd.5.smithi156.stdout:Apr 24 16:57:10 smithi156 podman[99893]: 2024-04-24 16:57:10.696238295 +0000 UTC m=+0.654615956 container died ef8a613c04c570e4ddb0fb834de339619818150858291c0680a458a7ca216f95 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T16:57:11.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:10.756+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:11.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:10.756+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:11.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:10 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:10.756+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:11.406 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.5.service' 2024-04-24T16:57:11.447 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:57:11.447 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-24T16:57:11.447 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-24T16:57:11.447 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.6 2024-04-24T16:57:11.528 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:11.263+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:11.528 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:11.263+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:11.528 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:11.263+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:11.806 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:11 smithi156 systemd[1]: Stopping Ceph osd.6 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:11.806 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:11.713+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:11.806 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:11.713+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:11.806 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:11.713+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:12.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:11.805+0000 7f56fe9db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:57:12.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:11.805+0000 7f56fe9db700 -1 osd.6 162 *** Got signal Terminated *** 2024-04-24T16:57:12.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:11 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:11.805+0000 7f56fe9db700 -1 osd.6 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:57:12.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:12.227+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:12.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:12.227+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:12.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:12.227+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:13.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:12.739+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:13.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:12.740+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:13.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:12 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:12.740+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:13.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:13.186+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:13.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:13.186+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:13.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:13.186+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:14.043 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:13.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:14.044 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:13.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:14.044 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:13 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:13.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:14.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:13.776+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:14.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:13.776+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:14.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:13 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:13.776+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:14.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:14.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:14.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:14.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:14.544 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:14 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:14.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:14.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:14.176+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:14.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:14.176+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:14.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:14.176+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:15.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:14.824+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:15.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:14.824+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:15.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:14 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:14.825+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:15.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:15.221+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:15.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:15.221+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:15.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:15.221+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:16.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:15.788+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:16.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:15.788+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:16.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:15 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:15.788+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:16.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:16.220+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:16.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:16.220+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:16.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:16.220+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:16.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:16.220+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:17.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:16.789+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:25.595602+0000 front 2024-04-24T16:56:25.595647+0000 (oldest deadline 2024-04-24T16:56:49.695620+0000) 2024-04-24T16:57:17.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:16.789+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:32.497418+0000 front 2024-04-24T16:56:32.497497+0000 (oldest deadline 2024-04-24T16:56:58.397187+0000) 2024-04-24T16:57:17.161 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:16 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6[85713]: 2024-04-24T16:57:16.789+0000 7f56fafcf700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:43.098194+0000 front 2024-04-24T16:56:43.098241+0000 (oldest deadline 2024-04-24T16:57:06.598148+0000) 2024-04-24T16:57:17.472 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:17 smithi156 podman[100000]: 2024-04-24 16:57:17.201634324 +0000 UTC m=+5.531694314 container died 2972333b52bf6b9962c3522b473fb6b20b7427f34235838f3db6b74cab712ffe (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T16:57:17.473 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:17.194+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:17.473 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:17.194+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:17.473 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:17.194+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:17.473 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:17 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:17.194+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:17.839 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:17 smithi156 podman[100000]: 2024-04-24 16:57:17.572223891 +0000 UTC m=+5.902283889 container cleanup 2972333b52bf6b9962c3522b473fb6b20b7427f34235838f3db6b74cab712ffe (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-24T16:57:17.840 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:17 smithi156 bash[100000]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6 2024-04-24T16:57:18.160 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:17 smithi156 podman[100019]: 2024-04-24 16:57:17.840047462 +0000 UTC m=+0.633671788 container remove 2972333b52bf6b9962c3522b473fb6b20b7427f34235838f3db6b74cab712ffe (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-24T16:57:18.451 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:18.168+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:18.451 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:18.168+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:18.451 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:18.168+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:18.451 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:18 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:18.168+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:18.451 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:18 smithi156 podman[100114]: 2024-04-24 16:57:18.182397961 +0000 UTC m=+0.106889763 container create 521e5e6da8781f1c2c59f185d4ac923549831223a49491471fd749a1e2e9deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-04-24T16:57:18.451 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:18 smithi156 podman[100114]: 2024-04-24 16:57:18.094121929 +0000 UTC m=+0.018613737 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:57:18.451 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:18 smithi156 podman[100114]: 2024-04-24 16:57:18.376388244 +0000 UTC m=+0.300880046 container init 521e5e6da8781f1c2c59f185d4ac923549831223a49491471fd749a1e2e9deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-24T16:57:18.452 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:18 smithi156 podman[100114]: 2024-04-24 16:57:18.379383026 +0000 UTC m=+0.303874828 container start 521e5e6da8781f1c2c59f185d4ac923549831223a49491471fd749a1e2e9deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-24T16:57:18.452 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:18 smithi156 podman[100114]: 2024-04-24 16:57:18.451355415 +0000 UTC m=+0.375847225 container attach 521e5e6da8781f1c2c59f185d4ac923549831223a49491471fd749a1e2e9deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-24T16:57:18.910 INFO:journalctl@ceph.osd.6.smithi156.stdout:Apr 24 16:57:18 smithi156 podman[100114]: 2024-04-24 16:57:18.668915989 +0000 UTC m=+0.593407798 container died 521e5e6da8781f1c2c59f185d4ac923549831223a49491471fd749a1e2e9deb0 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T16:57:19.238 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:19.126+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:19.238 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:19.126+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:19.238 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:19.126+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:19.238 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:19.126+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:19.264 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.6.service' 2024-04-24T16:57:19.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:57:19.341 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-24T16:57:19.341 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-24T16:57:19.342 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.7 2024-04-24T16:57:19.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 systemd[1]: Stopping Ceph osd.7 for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:20.115 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:19.665+0000 7f69c5089700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T16:57:20.115 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:19.665+0000 7f69c5089700 -1 osd.7 162 *** Got signal Terminated *** 2024-04-24T16:57:20.115 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:19 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:19.665+0000 7f69c5089700 -1 osd.7 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T16:57:20.115 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:20 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:20.115+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:20.115 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:20 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:20.115+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:20.116 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:20 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:20.115+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:20.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:20 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:20.115+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:21.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:21 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:21.103+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:21.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:21 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:21.103+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:21.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:21 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:21.103+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:21.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:21 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:21.103+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:22.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:22.077+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:22.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:22.077+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:22.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:22.077+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:22.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:22 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:22.077+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:23.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:23.122+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:23.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:23.122+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:23.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:23.122+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:23.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:23 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:23.122+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:24.046 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:23.556Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:24.046 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:23.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:24.046 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:23 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:23.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:24.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:24 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:24.110+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-24T16:56:26.286401+0000 front 2024-04-24T16:56:26.286453+0000 (oldest deadline 2024-04-24T16:56:50.386419+0000) 2024-04-24T16:57:24.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:24 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:24.110+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6814 osd.1 since back 2024-04-24T16:56:36.188227+0000 front 2024-04-24T16:56:36.188369+0000 (oldest deadline 2024-04-24T16:56:57.888130+0000) 2024-04-24T16:57:24.410 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:24 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:24.110+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6822 osd.2 since back 2024-04-24T16:56:44.190245+0000 front 2024-04-24T16:56:44.190219+0000 (oldest deadline 2024-04-24T16:57:10.090164+0000) 2024-04-24T16:57:24.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:24 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:24.110+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.70:6830 osd.3 since back 2024-04-24T16:56:52.291664+0000 front 2024-04-24T16:56:52.291598+0000 (oldest deadline 2024-04-24T16:57:15.791592+0000) 2024-04-24T16:57:24.411 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:24 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7[90677]: 2024-04-24T16:57:24.110+0000 7f69bd675700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.156:6804 osd.4 since back 2024-04-24T16:56:59.693310+0000 front 2024-04-24T16:56:59.693236+0000 (oldest deadline 2024-04-24T16:57:23.193266+0000) 2024-04-24T16:57:24.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:24.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:24.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:24.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi070:8443/api/prometheus_receiver\": dial tcp 172.21.15.70:8443: connect: connection refused" 2024-04-24T16:57:24.543 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:24 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:24.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi156.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.156:8443: connect: connection refused" 2024-04-24T16:57:25.366 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:25 smithi156 podman[100223]: 2024-04-24 16:57:25.108428122 +0000 UTC m=+5.542634513 container died e2d46d53eba41aefda13cc25602ec4a2a540e041ae973699664caa38a1931878 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-24T16:57:25.660 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:25 smithi156 podman[100223]: 2024-04-24 16:57:25.465745083 +0000 UTC m=+5.899951461 container cleanup e2d46d53eba41aefda13cc25602ec4a2a540e041ae973699664caa38a1931878 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-24T16:57:25.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:25 smithi156 bash[100223]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7 2024-04-24T16:57:26.091 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:25 smithi156 podman[100252]: 2024-04-24 16:57:25.741962009 +0000 UTC m=+0.628742341 container remove e2d46d53eba41aefda13cc25602ec4a2a540e041ae973699664caa38a1931878 (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-24T16:57:26.395 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:26 smithi156 podman[100346]: 2024-04-24 16:57:25.994714421 +0000 UTC m=+0.017969919 image pull 8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T16:57:26.395 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:26 smithi156 podman[100346]: 2024-04-24 16:57:26.100951178 +0000 UTC m=+0.124206665 container create 4551ac505637b28fc454a3b2cee8f75317485a56d99b0414b8d71e03567cef2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T16:57:26.395 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:26 smithi156 podman[100346]: 2024-04-24 16:57:26.320708469 +0000 UTC m=+0.343963953 container init 4551ac505637b28fc454a3b2cee8f75317485a56d99b0414b8d71e03567cef2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-24T16:57:26.395 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:26 smithi156 podman[100346]: 2024-04-24 16:57:26.323801927 +0000 UTC m=+0.347057412 container start 4551ac505637b28fc454a3b2cee8f75317485a56d99b0414b8d71e03567cef2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T16:57:26.395 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:26 smithi156 podman[100346]: 2024-04-24 16:57:26.395031506 +0000 UTC m=+0.418286994 container attach 4551ac505637b28fc454a3b2cee8f75317485a56d99b0414b8d71e03567cef2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-24T16:57:26.661 INFO:journalctl@ceph.osd.7.smithi156.stdout:Apr 24 16:57:26 smithi156 podman[100346]: 2024-04-24 16:57:26.614160354 +0000 UTC m=+0.637415827 container died 4551ac505637b28fc454a3b2cee8f75317485a56d99b0414b8d71e03567cef2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-0937-5904d29, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-24T16:57:27.189 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@osd.7.service' 2024-04-24T16:57:27.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:57:27.227 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-24T16:57:27.227 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-24T16:57:27.227 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a 2024-04-24T16:57:27.559 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 systemd[1]: Stopping Ceph prometheus.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-24T16:57:27.818 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-24T16:57:27.819 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.559Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-24T16:57:27.819 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.560Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-24T16:57:27.819 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.560Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-24T16:57:27.819 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a[61266]: ts=2024-04-24T16:57:27.560Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-24T16:57:27.819 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 podman[100452]: 2024-04-24 16:57:27.569752777 +0000 UTC m=+0.120070862 container died f240eb7d50fac207ebc3c6697f3b67c79ad9eddca1d1ccc66ef4ffa9e6e11ba3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 podman[100452]: 2024-04-24 16:57:27.917260746 +0000 UTC m=+0.467578830 container cleanup f240eb7d50fac207ebc3c6697f3b67c79ad9eddca1d1ccc66ef4ffa9e6e11ba3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:28.160 INFO:journalctl@ceph.prometheus.a.smithi156.stdout:Apr 24 16:57:27 smithi156 bash[100452]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-prometheus-a 2024-04-24T16:57:28.335 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-be31be9a-0255-11ef-bc93-c7b262605968@prometheus.a.service' 2024-04-24T16:57:28.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T16:57:28.367 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-24T16:57:28.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid be31be9a-0255-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T16:57:30.293 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:29 smithi070 systemd[1]: Stopping Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:30.294 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 podman[129745]: 2024-04-24 16:57:30.071988769 +0000 UTC m=+0.105735736 container died 88827079f0f687430af3bf1d30651e16a7280562e95c51f313011fe34b3d57e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:30.778 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 podman[129745]: 2024-04-24 16:57:30.511209372 +0000 UTC m=+0.544956339 container cleanup 88827079f0f687430af3bf1d30651e16a7280562e95c51f313011fe34b3d57e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:30.779 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 bash[129745]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a 2024-04-24T16:57:31.043 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 podman[129759]: 2024-04-24 16:57:30.779063728 +0000 UTC m=+0.711376843 container remove 88827079f0f687430af3bf1d30651e16a7280562e95c51f313011fe34b3d57e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:31.043 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:57:31.043 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-24T16:57:31.043 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 systemd[1]: Stopped Ceph node-exporter.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:57:31.043 INFO:journalctl@ceph.node-exporter.a.smithi070.stdout:Apr 24 16:57:30 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 3.144s CPU time. 2024-04-24T16:57:31.401 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:31 smithi070 systemd[1]: Stopping Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:31.401 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:31 smithi070 ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a[76644]: ts=2024-04-24T16:57:31.340Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-24T16:57:31.658 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:31 smithi070 podman[129864]: 2024-04-24 16:57:31.4016741 +0000 UTC m=+0.153360246 container died a84220aabb99879cecf9bd73d86bf9b4f3fe918426a17a37d44c51128e2d8f97 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:32.016 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:31 smithi070 podman[129864]: 2024-04-24 16:57:31.757760124 +0000 UTC m=+0.509446270 container cleanup a84220aabb99879cecf9bd73d86bf9b4f3fe918426a17a37d44c51128e2d8f97 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:32.017 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:31 smithi070 bash[129864]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a 2024-04-24T16:57:32.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:32 smithi070 podman[129879]: 2024-04-24 16:57:32.017073633 +0000 UTC m=+0.609943324 container remove a84220aabb99879cecf9bd73d86bf9b4f3fe918426a17a37d44c51128e2d8f97 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-24T16:57:32.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:32 smithi070 podman[129879]: 2024-04-24 16:57:32.10100692 +0000 UTC m=+0.693876610 volume remove d8fe49e43498d6b2af14cde3c8e469e647f89528c8bce001794b8c63dc913f00 2024-04-24T16:57:32.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:32 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-24T16:57:32.293 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:32 smithi070 systemd[1]: Stopped Ceph alertmanager.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:57:32.294 INFO:journalctl@ceph.alertmanager.a.smithi070.stdout:Apr 24 16:57:32 smithi070 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@alertmanager.a.service: Consumed 1.698s CPU time. 2024-04-24T16:57:55.220 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid be31be9a-0255-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T16:57:57.089 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:56 smithi156 systemd[1]: Stopping Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:57.089 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:56 smithi156 podman[100849]: 2024-04-24 16:57:56.904615287 +0000 UTC m=+0.100336622 container died 9770cba17afbc0ae7473c0ec0a024192a3763d700d7fd1081274b89fa4ed1103 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:57:57.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:57 smithi156 podman[100849]: 2024-04-24 16:57:57.235517134 +0000 UTC m=+0.431238478 container cleanup 9770cba17afbc0ae7473c0ec0a024192a3763d700d7fd1081274b89fa4ed1103 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:57:57.410 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:57 smithi156 bash[100849]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b 2024-04-24T16:57:57.678 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:57 smithi156 podman[100863]: 2024-04-24 16:57:57.428313426 +0000 UTC m=+0.527872440 container remove 9770cba17afbc0ae7473c0ec0a024192a3763d700d7fd1081274b89fa4ed1103 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-24T16:57:57.678 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:57 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-24T16:57:58.114 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:57 smithi156 systemd[1]: Stopping Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968... 2024-04-24T16:57:58.115 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:57 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-24T16:57:58.115 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:57 smithi156 systemd[1]: Stopped Ceph node-exporter.b for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:57:58.115 INFO:journalctl@ceph.node-exporter.b.smithi156.stdout:Apr 24 16:57:57 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 3.148s CPU time. 2024-04-24T16:57:58.382 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=server t=2024-04-24T16:57:58.114586623Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-24T16:57:58.382 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=ticker t=2024-04-24T16:57:58.114981187Z level=info msg=stopped last_tick=2024-04-24T16:57:50Z 2024-04-24T16:57:58.382 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a[63485]: logger=tracing t=2024-04-24T16:57:58.114937631Z level=info msg="Closing tracing" 2024-04-24T16:57:58.382 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 podman[100981]: 2024-04-24 16:57:58.12481977 +0000 UTC m=+0.107535417 container died 135413e06c8551842ce753383a4490a5cbba6106687257f3abeb1c1a137da745 (image=quay.io/ceph/grafana:9.4.12, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-24T16:57:58.660 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 podman[100981]: 2024-04-24 16:57:58.447564923 +0000 UTC m=+0.430280574 container cleanup 135413e06c8551842ce753383a4490a5cbba6106687257f3abeb1c1a137da745 (image=quay.io/ceph/grafana:9.4.12, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-24T16:57:58.660 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 bash[100981]: ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a 2024-04-24T16:57:58.660 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 podman[100995]: 2024-04-24 16:57:58.632392853 +0000 UTC m=+0.507433898 container remove 135413e06c8551842ce753383a4490a5cbba6106687257f3abeb1c1a137da745 (image=quay.io/ceph/grafana:9.4.12, name=ceph-be31be9a-0255-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-24T16:57:59.160 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-24T16:57:59.160 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 systemd[1]: Stopped Ceph grafana.a for be31be9a-0255-11ef-bc93-c7b262605968. 2024-04-24T16:57:59.161 INFO:journalctl@ceph.grafana.a.smithi156.stdout:Apr 24 16:57:58 smithi156 systemd[1]: ceph-be31be9a-0255-11ef-bc93-c7b262605968@grafana.a.service: Consumed 11.910s CPU time. 2024-04-24T16:58:10.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T16:58:10.532 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T16:58:10.558 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-24T16:58:10.560 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778/remote/smithi070/crash 2024-04-24T16:58:10.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/crash -- . 2024-04-24T16:58:10.704 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T16:58:10.704 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2024-04-24T16:58:10.706 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778/remote/smithi156/crash 2024-04-24T16:58:10.707 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/crash -- . 2024-04-24T16:58:10.750 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/be31be9a-0255-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T16:58:10.750 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2024-04-24T16:58:10.751 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-24T16:58:10.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-24T16:58:10.783 INFO:teuthology.orchestra.run.smithi070.stdout:2024-04-24T16:42:01.554619+0000 mon.a (mon.0) 1078 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:58:10.783 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-24T16:58:10.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[SEC\]' /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-24T16:58:10.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]' /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-24T16:58:10.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[WRN\]' /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-24T16:58:10.987 INFO:teuthology.orchestra.run.smithi070.stdout:2024-04-24T16:42:01.554619+0000 mon.a (mon.0) 1078 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state'] (CEPHADM_FAILED_DAEMON) 2024-04-24T16:58:10.988 INFO:tasks.cephadm:Compressing logs... 2024-04-24T16:58:10.989 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T16:58:11.030 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:: No such file or directory 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi070.vhdldq.log 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi070.vhdldq.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr: 75.9% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi070.vhdldq.log.gz 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-24T16:58:11.053 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-24T16:58:11.054 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-24T16:58:11.054 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-24T16:58:11.058 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T16:58:11.058 INFO:teuthology.orchestra.run.smithi156.stderr:: No such file or directory 2024-04-24T16:58:11.058 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi156.pmtzwp.log 2024-04-24T16:58:11.058 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi156.pmtzwp.log: /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log: 76.1% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi156.pmtzwp.log.gz 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-24T16:58:11.059 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T16:58:11.060 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.6.log: 89.1% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T16:58:11.060 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.0.log: 93.9% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T16:58:11.060 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-24T16:58:11.062 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T16:58:11.065 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T16:58:11.065 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-24T16:58:11.065 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-volume.log: 92.2% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T16:58:11.066 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.c.log: 94.4% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T16:58:11.066 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T16:58:11.066 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.a.log: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T16:58:11.078 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/tcmu-runner.log 2024-04-24T16:58:11.081 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T16:58:11.081 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.cephadm.log: 95.2% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T16:58:11.084 INFO:teuthology.orchestra.run.smithi156.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T16:58:11.090 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T16:58:11.091 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/tcmu-runner.log: 89.1% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/tcmu-runner.log.gz 2024-04-24T16:58:11.119 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T16:58:11.170 INFO:teuthology.orchestra.run.smithi156.stderr: 89.8% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-24T16:58:11.575 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-24T16:58:11.644 INFO:teuthology.orchestra.run.smithi156.stderr: 92.5% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-24T16:58:11.957 INFO:teuthology.orchestra.run.smithi070.stderr: 92.3% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-24T16:58:12.421 INFO:teuthology.orchestra.run.smithi070.stderr: 93.3% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-24T16:58:12.830 INFO:teuthology.orchestra.run.smithi156.stderr: 93.2% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-24T16:58:12.869 INFO:teuthology.orchestra.run.smithi070.stderr: 93.5% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-24T16:58:12.962 INFO:teuthology.orchestra.run.smithi070.stderr: 91.0% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-24T16:58:12.965 INFO:teuthology.orchestra.run.smithi156.stderr: 93.8% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-24T16:58:13.085 INFO:teuthology.orchestra.run.smithi070.stderr: 93.4% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-24T16:58:13.090 INFO:teuthology.orchestra.run.smithi156.stderr: 93.4% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-24T16:58:13.141 INFO:teuthology.orchestra.run.smithi156.stderr: 93.6% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-24T16:58:13.143 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-04-24T16:58:13.143 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m2.098s 2024-04-24T16:58:13.143 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m8.219s 2024-04-24T16:58:13.143 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.332s 2024-04-24T16:58:13.686 INFO:teuthology.orchestra.run.smithi070.stderr: 93.4% -- replaced with /var/log/ceph/be31be9a-0255-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-24T16:58:13.688 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-24T16:58:13.688 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m2.646s 2024-04-24T16:58:13.688 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m10.837s 2024-04-24T16:58:13.688 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m0.387s 2024-04-24T16:58:13.688 INFO:tasks.cephadm:Archiving logs... 2024-04-24T16:58:13.688 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778/remote/smithi070/log 2024-04-24T16:58:13.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T16:58:15.302 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778/remote/smithi156/log 2024-04-24T16:58:15.303 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T16:58:16.527 INFO:tasks.cephadm:Removing cluster... 2024-04-24T16:58:16.527 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid be31be9a-0255-11ef-bc93-c7b262605968 --force 2024-04-24T16:58:16.815 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid be31be9a-0255-11ef-bc93-c7b262605968 --force 2024-04-24T16:58:17.105 INFO:tasks.cephadm:Removing cephadm ... 2024-04-24T16:58:17.106 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T16:58:17.121 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T16:58:17.137 INFO:tasks.cephadm:Teardown complete 2024-04-24T16:58:17.137 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T16:58:17.151 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T16:58:17.151 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T16:58:17.164 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T16:58:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-24T16:58:17.180 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T16:58:17.180 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-24T16:58:17.180 INFO:teuthology.orchestra.run.smithi070.stdout:^- hv01.front.sepia.ceph.com 2 6 377 29 +1999us[+1998us] +/- 585ms 2024-04-24T16:58:17.180 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -232us[ -234us] +/- 65ms 2024-04-24T16:58:17.180 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +312us[ +311us] +/- 79ms 2024-04-24T16:58:17.180 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:58:17.190 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-24T16:58:17.194 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T16:58:17.194 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-24T16:58:17.195 INFO:teuthology.orchestra.run.smithi156.stdout:^- hv01.front.sepia.ceph.com 2 6 377 30 +2012us[+2000us] +/- 585ms 2024-04-24T16:58:17.195 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -218us[ -231us] +/- 64ms 2024-04-24T16:58:17.195 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +319us[ +319us] +/- 79ms 2024-04-24T16:58:17.195 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T16:58:17.195 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T16:58:17.234 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T16:58:17.234 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T16:58:17.243 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T16:58:17.327 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T16:58:17.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T16:58:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:58:17.428 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T16:58:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:58:17.455 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T16:58:17.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T16:58:17.474 INFO:teuthology.task.internal:Duration was 2829.272632 seconds 2024-04-24T16:58:17.475 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T16:58:17.484 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T16:58:17.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T16:58:17.487 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T16:58:17.520 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T16:58:17.531 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T16:58:17.698 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T16:58:17.698 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-04-24T16:58:17.699 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T16:58:17.723 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T16:58:17.723 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T16:58:17.759 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T16:58:17.760 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T16:58:17.765 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T16:58:17.825 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T16:58:17.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T16:58:17.830 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T16:58:18.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T16:58:18.415 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T16:58:18.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T16:58:18.438 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T16:58:18.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T16:58:18.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:58:18.481 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:58:18.500 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-04-24T16:58:18.527 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-04-24T16:58:18.558 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:58:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:58:18.587 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T16:58:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:58:18.612 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T16:58:18.623 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T16:58:18.624 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778/remote/smithi070 2024-04-24T16:58:18.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T16:58:18.679 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778/remote/smithi156 2024-04-24T16:58:18.680 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T16:58:18.722 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T16:58:18.722 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T16:58:18.725 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T16:58:18.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T16:58:18.781 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T16:58:18.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T16:58:18.792 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T16:58:18.792 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T16:58:18.794 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T16:58:18.806 INFO:teuthology.orchestra.run.smithi070.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 16:58 /home/ubuntu/cephtest 2024-04-24T16:58:18.825 INFO:teuthology.orchestra.run.smithi156.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 16:58 /home/ubuntu/cephtest 2024-04-24T16:58:18.826 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T16:58:18.837 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T16:58:18.868 INFO:teuthology.nuke:Checking targets against current locks 2024-04-24T16:58:18.897 DEBUG:teuthology.nuke:shortname: smithi070 2024-04-24T16:58:18.898 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T16:58:18.921 DEBUG:teuthology.nuke:shortname: smithi156 2024-04-24T16:58:18.921 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T16:58:18.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 16:05:24.305906', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDmJfBoZw+B1CZPo3ctZgRjX+PElSCyGtxZy011gg5DYYmbCARINLPaU2BHCI5ySvixRHiNMxl+eLMyyXOWKeDM='} 2024-04-24T16:58:18.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671778', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 16:05:24.304550', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPn+3/er7ip06BTa3GEvE4tISGx6yi1N/Lc7LwpQJH3ELMqK+xGcZUt7/Yriq2YQNSLq6oIW1WyotbW62qtSStU='} 2024-04-24T16:58:18.995 INFO:teuthology.orchestra.console.smithi070:Power off 2024-04-24T16:58:18.995 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T16:58:19.024 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-04-24T16:58:19.024 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T16:58:19.047 INFO:teuthology.orchestra.console.smithi156:Power off 2024-04-24T16:58:19.047 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T16:58:19.061 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-24T16:58:19.071 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2024-04-24T16:58:19.072 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T16:58:19.103 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-04-24T16:58:23.063 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T16:58:23.104 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T16:58:23.173 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-24T16:58:23.216 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-04-24T16:58:27.179 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T16:58:27.217 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T16:58:27.292 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-04-24T16:58:27.330 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2024-04-24T16:58:27.393 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-04-24T16:58:27.431 INFO:teuthology.orchestra.console.smithi156:Power off completed 2024-04-24T16:58:27.537 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2829.27263212204 failure_reason: '"2024-04-24T16:42:01.554619+0000 mon.a (mon.0) 1078 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) [''daemon iscsi.foo.smithi070.gcoqkr on smithi070 is in unknown state''] (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_adking@teuthology success: false 2024-04-24T16:58:27.538 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T16:58:27.662 INFO:teuthology.run:FAIL